shithub: neoventi

--- a/disk.c

+++ b/disk.c

@@ -68,7 +68,7 @@

 	bucket -= sect.start;

 	buf = malloc(sect.blocksize);

 	if(buf == nil)

-		sysfatal("OOM");

+		sysfatal("OOM, blocksize: %d", sect.blocksize);

 	if(pread(sect.fd, (char*)buf, sect.blocksize, sect.blockbase + (bucket << sect.blocklog)) != sect.blocksize)

 		sysfatal("Failed to read bucket");

 	bentries = U16GET(buf);

--- a/main.c

+++ b/main.c

@@ -1,6 +1,7 @@

 #include <u.h>

 #include <libc.h>

 #include <bio.h>

+#include <thread.h>

 #include "neoventi.h"

 void

@@ -25,7 +26,7 @@

 void

-main(int argc, char **argv)

+threadmain(int argc, char **argv)

 	parseargs(argc, argv);

 	print("Initializing neoventi build 3... ");

--- a/notebook

+++ b/notebook

@@ -356,3 +356,46 @@

 	free(buf);

 	return 1;

+<date Sun Dec 24 19:09:00 EST 2023

+Okay! Everything works and has been partly cleaned up and refactored. Needs more of that, but getting the write path working is a higher priority right now.

+Probably going to want to buffer writes, and actually handle Tsync as a flush call. If I'm going to want a server-side cache as well, I should probably have them unified, and have a separate set of dirty entries? The question is, what layer do I put the cache at?

+I could do a disk-level cache, which tracks data by address on disk, with a dirty bit on each entry - this would be a singular cache which handles both index and arena data. IIRC this would be akin to venti's dcache, and I'm not sure I want that.

+The index cache is the most useful, though if it does not get flushed, we need to do a scan on startup to populate it the same way venti does, and this has been one of the behaviors that has irked me. Blocking until all data is available is reasonable in this situation, but getting *into* that situation is kinda crazy.

+One of the major annoyances with original venti is how poorly it utilizes the storage disk. While writing data, it will take a minute before it starts flushing _any_ of it out. We should be trying to write data more immediately than that.

+On the other hand, always writing immediately may not be ideal? but eh, I'd rather try anyways and then see how it works out. Venti is WORM, so normal patterns fall into two general categories in my experience:

+• File system running on top

+	- This flushes periodically

+	- Most of the time, it will be totally idle

+	- When it writes, it writes all data at once

+• One-time upload of data

+	- Archives, vacballs

+	- This should also just fire data at us full-speed

+In both situations, we should expect to have a constant stream of data when we're writing. We have two streams of data to write:

+• The log

+	• This is written linearly, chunk after chunk after chunk

+	• Each chunk has a checksum, but calculating those is trivial

+• The index

+	• This writes to totally random parts of the disk

+		• _scatter storage addressing_, after all!

+Buffering the log might be reasonable, since it allows for writing larger blocks at a time.

+but... why speculate when I can measure? Start by just writing data the instant it comes in. If that's not fast enough, look into using multiple procs for flushing data. If that's not fast enough, look into other options. Overengineering is Bad™.

+Just double checked venti and it looks like it has one writer per partition, so it shouldn't be terribly hard to beat its performance without even resorting to multithreading - just by having a better implementation of the single-threaded writing.

+Caching / prefetching / some such system will definitely be needed to win on the read side - or, well, *probably* be needed. Parallelized readers might be sufficient, though that may require changes to libventi / vacfs / etc to enable multiple reads in flight at a time. But again, measure before designing.

+Worst case, neoventi's prototype has demonstrated that I _can_ do this, even if I have to change the design a bunch more. And, I kinda want to do the final implementation in Hare anyways.

--- a/server.c

+++ b/server.c

@@ -1,6 +1,7 @@

 #include <u.h>

 #include <libc.h>

 #include <bio.h>

+#include <thread.h>

 #include "neoventi.h"

 // Handles an error on `conn` handling client request `tbuf`

@@ -120,9 +121,15 @@

 		vterr(conn, buf, "failed to rhello: %r");

+typedef struct {

+	int ctl;

+	char *dir;

+} ClientHandle;

 static void

-handle(int ctl, char *dir)

+handleproc(void *hnd)

+	ClientHandle *h = hnd;

 	char buf[MaxPacketSize];

 	VtConn conn;

 	switch(setjmp(conn.bounce)){

@@ -132,12 +139,13 @@

 		fprint(2, "abandoning client: %r\n");

 	case 2:

 		close(conn.fd);

+		free(h);

 		return;

 	default:

 		fprint(2, "internal error: unexpected bounce code\n");

 		break;

-	conn.fd = accept(ctl, dir);

+	conn.fd = accept(h->ctl, h->dir);

 	if(conn.fd < 0)

 		vterr(conn, nil, "failed to accept connection: %r");

 	vthello(conn, buf);

@@ -148,8 +156,18 @@

 			return;

+static void

+handle(int ctl, char *dir)

+{

+	ClientHandle *handle = malloc(sizeof(ClientHandle));

+	handle->ctl = ctl;

+	handle->dir = dir;

+	proccreate(handleproc, handle, mainstacksize);

+}

 void

 serve(char *addr)

@@ -160,7 +178,6 @@

 		sysfatal("%r");

 	procsetname("neoventi/server");

 	for(ctl = listen(adir, dir); ctl >= 0; ctl = listen(adir, dir)){

-		// TODO: handle the client in a worker, allow multiple clients

 		handle(ctl, dir);

 		close(ctl);