Transitioning Cyberix from traditional databases to a flat file-based design for the future

1 replies
1 attachments
Started 10h ago

28 days later.mp4
28 days later.mp4
I am looking to design a filesystem-as-database architecture for a high-anonymity, low-trust forum that needs to be accessible from about half a dozen protocols.

Why?
- Don't want to write special code to access database formats
- Flat files are simpler, more 'native' and easier to manipulate
- Backups and restorations become as simple as an rsync or a cp -r
- Infact, backups in general should become more flexible and easier to conduct

Concerns:
Search (inverted index could solve this)
Long threads and pagination slowing down performance (maintaining per-page caches that are rebuilt on write could solve this?)
Concurrency issues (All requests sent to special requests folder that is iterated upon by an asynchronous worker that also acts as our antispam worker that fulfills the logic that the PHP worker currently fulfills
Mod panel intricacies that involve scanning and searching lots of posts for little bits of metadata (Inverted index??

Ideally:
Self managed and self maintained by a worker. I would rather not manually code cache rebuilds or indexes or anything complex.
The core of the forum should be standardized across all protocols. It reads and writes data to what is reasonable.
The workers control and manage everything. The protocols simply make requests.
The Mod Panel remains on HTTP only, though

Some examples:
1 - HTTP
Write post. Click post.
PHP does whatever it needs to do to write a file to /requests with the relevant metadata. Worker takes care of it from there
Click thread. Read thread.
Uhhhhhh...

This is an idea that I am going to implement and I will implement it but it's not exactly thorough and there isn't a complete design or architecture that I've settled on, yet.
Attachments:
Replies: >>18054

[AutoMod] action=keep R:10 E:10 N:9 C:9 | The user clearly outlines a complex architectural goal. The reasoning is logical and directly addresses the stated needs. It's a solid, high-level technical proposal.

>>18052
wat

[US-TX]

Reply

Posting anonymously. Your IP address will be recorded for rate limiting purposes.





Max 10MB per file. Allowed: images, videos, audio, PDF, text, zip