I Need feedback on the archive category

Hey All,

I would like some feedback on the archive category. but first…

What is the Archive category? It is a new category that will one day contain the ENTIRE old www.chaos.dwarf.com forum content. I have been writing a piece of software that does A LOT to make sure the data stays as consistent as possible.

the tool does the following:

  • It takes a part (this is configurable) of ALL the posts ever made on the old forum (0.25 million) and loops through them chronologically.
  • It checks if the post’s category already exists on discourse, if not it is created
  • it checks if there is a topic for this post, if not it creates the topic (the post then technically is a topic)
  • If there is a topic it creates a reply in the topic
    → the message is sanitized, meaning the MyBB markup tags that do not work on discourse are removed/replaced.
    → the message is scanned for ALL links that either refer to the old forum (to another topic or something) OR to images that were hosted on the forum itself. These links are stored in a database for later reprocessing, since some posts do not yet exist at this point.
    → The original forums original username is prefixed to the message for continuity’s sake. Usernames are matched by a High probability algorithm so “Pyro Stick” and “Pyrostick” can be matched. This MIGHT not always be 100% correct.
    → Posts by Users that do not exist on this forum are made by the “System” user.
  • The creation of the post on discourse is logged in a database so i can link the old post id to the new discourse post at any time AND so i always know what post was last processed.
  • Whenever i feel like it, or maybe ill do it automatically after each bulk transfer, or whenever :man_shrugging: Ill take those url’s that i saved in the database and do a lookup for the Discourse version of that page. If it’s found the post is updated and saved. The record is marked as complete.
    → This will be extended to also upload the missing images that are physically on my pc but only after storage on discourse has been taken into account.

Now… This is a lot of work to check, and not everything is fixable. A dead external link will still be dead. Some MyBB functionality (wiki linking and such) will not work if the old MyBB plugin is used) and so forth.

If there is anyone who has some spare time would be willing to do some comparing and checking of the archive and the old forums version that would be GREAT!

these posts have had some of their URL’s updated by the replacement code.

https://discourse.chaos-dwarfs.com/t/archive-compiled-conversions/2697?u=michaelx
https://discourse.chaos-dwarfs.com/t/archive-compiled-gallery/2699
https://discourse.chaos-dwarfs.com/t/archive-chaos-dwarfs-online-rules-and-forum-primer/2742/1
https://discourse.chaos-dwarfs.com/t/archive-introductions-new-members-post-here/2721/6
https://discourse.chaos-dwarfs.com/t/archive-compiled-conversions/2697/2

for example the first Xander link in the fist post, the pulper link in the second post and the links in the last two posts.

Comparing both forums is fairly simple

open the post on Discourse you want to check
image
On the old forum you would go to the “Modeling Ideas and Advice” category, and look for the “Compiled Conversions” thread (probably on the last page, but a search on the topic name works well too. )

Any and ALL feedback is very welcome!

4 Likes

What i’m i mostly looking for?

  • External links that do not show a preview, but when clicked do still work
  • Weird text or markup syntax
  • missing bits
1 Like

Think of all the cool inspiration we’re going to find looking through this stuff! I bet @Admiral and @tjub can point us to cool old threads and blogs, too.

2 Likes

And important to note is, i only transfered 230 posts, so A LOT of links still point to the old forum. That should dissapear when the tool has processed more.

I’ve been taking a look at some of the really old stuff thats been appearing anyway, so it’s not really any hassle to cross-reference if anything looks looks bit wrong.

Is it better to let you know on discord or here so there’s a more long-term reference?

1 Like

Here is fine to keep track of it. Thanks mate!

1 Like

I’ve found 1 issue myself.
When there is a quote block at the start of a message it does not work because i prefixe the old username to it.

Like so

MichaelX: [quote=MichaelX]
When there is a quote block at the start of a message it does not work because i prefixe the old username to it.
[/quote]

vs

MichaelX:

Ive added a new line after the username to fix this.

1 Like

I can lend a hand with writing “about” sections for different sections. Probably won’t have the spare time for the cross referencing stuff you require mate

No worries. I can check if i can take the old forum descriptions :smiley:,
Edit; good shout though

1 Like

@Oxymandias implemented.

see the https://discourse.chaos-dwarfs.com/c/archive/archive-staff/177 and https://discourse.chaos-dwarfs.com/c/archive-army-lists/178 topics.

After all feedback has been processed ill delete nuke the entire archive category and everything in it and start over. Then all the feedback will show.

1 Like

@MichaelX is the archive staff section open to public? Probably best it’s staff only like the current one. Staff organisation from 2007 is likely to be okay but especially as they get more recent we’d probably want them all locked down. Things like login sections and staff decisions about sensitive things may come up that we wouldn’t want to show up on Google haha

1 Like

Yes and agreed. Either I don’t move staff over (exclude it) or lock it down.

1 Like

made it staff only for now, manually

Jupp, good catch!

Also, just hade a quick look and found Kyte’s old stuff there… Such a trip down memory lane!

2 Likes

Right…

If nothing pops up in the following days ill delete everything thats created up to this point and start the actual import process next monday.

2 Likes

Exciting! From what I can tell it looks good, and alot more content than I had thought seems to have survived the years. :slight_smile:

There are a few formatting errors based on syntax from the old board that no longer has function and therefore leaves artefacts.

The most important one I’ve seen that reoccurs is
 Â
Which, I think maybe is a tab or bullet point?
You also see a lot of html-style markup, [list] [/list] and the like. There’s also some that’s [] which must be Discourse interpreting [ * ] * (without the spaces)

Can you give me post examples?

I built the Image upload fuction for the direct upload stuff.

https://discourse.chaos-dwarfs.com/t/archive-xanders-chaos-dwarf-kollossus-conversion-updated-march-13th/2696/1

Its not yet 100% but it seems to work fine.

1 Like

I don’t see any pictures from the links, but maybe I’m not supposed to yet?