A four colour any-pixel-addressable video mode for the Master System

Wednesday, 4th August 2021

The video display processor in the Sega Master System is derived from the TMS9918A and provides a number of different screen modes. These modes are mostly character-based and generate the picture that is displayed on your TV based on two blocks of data in video RAM: the name table, which specifies which character appears in each cell on the screen and the pattern generator which contains the pixel data for each character.

TMS9918A "Text" mode

The "Text" mode is probably the most straightforward example. Only two colours can be displayed on the screen; a foreground and a background colour, both selected from a fixed palette of 15 colours. The name table is 40 cells wide and 24 tall. Each cell, or text position, is a single byte that refers to one of 256 patterns in the pattern generator, so the complete name table is 960 bytes long. Each character is displayed as 6×8 pixels but the data in the pattern generator is eight pixels wide, and each row is stored as a single byte with a set bit selecting the foreground colour and a cleared bit selecting the background colour (the two least-significant bits are ignored). Each character pattern therefore takes up 8 bytes of VRAM, or 2,048 bytes (2KB) for a complete set of 256.

The "Graphics I" mode changes the character size to be the full 8×8 pixels at the expense of the number of characters that can be displayed per line; this is reduced from 40 to 32, so the name table is now 32×24 and 768 bytes long. The pattern generator is the same size as before and you can still only display 256 unique patterns on-screen at a time, but there is at least some provision for colour. The top five bits of the character/pattern number are used as an index into a 32-byte colour table, where each colour table entry contains two four-bit values corresponding to the foreground and background colours for the pattern. This means that patterns 0-7 share the same two colours, patterns 8-15 have the next set of two colours, 16-23 after that and so on; 256 patterns, in groups of eight with a different pair of colours assigned to each of those 32 groups.

TMS9918A "Graphics II" mode

Fortunately, the TMS9918A is a revised version of the original TMS9918 which adds the much more useful "Graphics II" mode. The name table is the same as before, 32×24 cells of one byte each, and the patterns are still 8×8 pixel bitmaps. However the pattern generator table is now three times the size, storing 768 unique 8×8 patterns in 6KB. To be able to address all 768 patterns when you can only store values from 0-255 in the name table the screen is divided into three sections; the top third references the first 2KB of the pattern generator (patterns 0-255), the middle third references the second 2KB of the pattern generator (patterns 256-511) and the bottom third references the last 2KB of the pattern generator (patterns 512-767). This allows every single character position in the nametable to reference a unique pattern, and therefore have a bitmap that covers the entire screen area.

The colour table has also been significantly improved. Instead of one pair of colours for each group of eight patterns, each pattern now gets its own table of colour pairs, one per row. This doesn't allow every pixel on the screen to have its own colour, but you can at least give each 8×1 region its own pair of colours. This does lead to attribute clash when drawing; you can see this in the screenshot of the cones above, where the colour from some lines bleeds into the colour set for nearby lines if they happen to pass through the same 8×1 pixel block.

Very few Master System games use the TMS9918A video modes (F16 Fighting Falcon seems to be about the only concrete example), instead using a new video mode added specifically for the Master System. This has a number of nice new features (such as hardware scrolling and two user-definable 16-colour palettes instead of a fixed 15-colour one), but it's the name table and pattern generator that's of main interest. Instead of having a separate bitmap for the pattern and its colour table, each pattern is now a 32 byte object where each 8 pixel row is encoded as four bytes, one byte for each bit of the four-bit colour palette index. That is, if you wanted to store the top row of a pattern where the leftmost pixel was set to palette index 1, the rightmost pixel was set to palette index 15 and everything else was left as 0 it would be stored like this:

.db %10000001
.db %00000001
.db %00000001
.db %00000001

In practice you are limited to 448 patterns, which is not enough to cover the screen. As you will generally start from a blank screen and then add to it, the name table could be filled with blanks by default and then as you draw over the screen of it new patterns could be allocated as required. This is how the graphics currently work in the 16-colour Master System video mode, and it allows for colourful drawings with no colour clash as long as you don't fill too much of the screen.

Cautious use of the 16-colour Master System mode

That's the mode the sphere above is rendered in; you can see there's fine detail at its North pole without becoming subject to colour clash and it doesn't fill so much of the screen that we ran out of unique patterns to allocate. As it would be very bad to run out of patterns when attempting to draw text the character set (the 96 characters from 32-127) are permanently allocated as unique patterns which further reduces us to 352 tiles for graphics output.

It's possible to constrain the graphics viewport via VDU 24 to a smaller region which would ensure you never needed more than 352 tiles, as long as your new viewport was properly aligned to the 8×8 grid. For example, 22×16=352 which means that you could have a 176×128 pixel graphics window that you could draw to without risking running out of patterns.

   
Colour clash in Graphics II (left) versus running out of tiles in Master System Mode (right)

The two screenshots above show the trade-off between colour clash in Graphics II and corruption due to running out of patterns in the 16-colour Master System graphics mode, both caused by running this program:

 10 INPUT "Mode",M%
 20 MODE M%
 30 FOR I%=0 TO 47
 40   GCOL 0,I%
 50   X%=(I%/47)*1279
 60   Y%=(I%/47)*959
 70   MOVE X%,0
 80   DRAW 1279,Y%
 90   DRAW 1279-X%,959
100   DRAW 0,959-Y%
110   DRAW X%,0
120 NEXT I%
130 REPEAT:UNTIL INKEY(0)=-1
140 REPEAT:UNTIL INKEY(0)<>-1

It is possible to completely avoid colour clash if you use the Graphics II mode and simply never use any colour commands, but only having two colours is a bit drab. With a bit of tinkering I came up with another graphics mode that is still using the Master System mode, but reduces the number of colours down to 4. Quite fittingly, a lot of BBC Micro screen modes only supported four physical colours, so I set this new mode to use the same black/red/yellow/white palette by default:

Four colours on-screen at once, no clash and no corruption

In fact, the palette is the whole key to this solution. You may recall that the Master System mode supports two 16-colour palettes. Usually one is assigned as the "background" palette and the other as the "sprite" palette, however each entry in the name table contains a flag that lets you specify which palette the selected pattern should be shown with. In this new video mode, the top half of the screen is assigned to patterns 0-383 using the "background" palette, and the bottom half of the screen is assigned to the same set of patterns 0-383 but to use the "sprite" palette. The palette is then filled with four colours, like this:

Using all 32 palette entries to display four colours on screen simultaneously

The top row is the "background" palette and shows the four desired colours repeating in order four times. The bottom row is the "sprite" palette, and shows that each of the four colours is stretched to fill four consecutive palette entries. This use of repetition effectively turns two bits in each pattern's bitplanes into "don't care" values, depending on which palette is selected.

For example, assume we're in a pattern assigned to the "background" (top) palette and we want to display red. This is colour %01, but as the red entry is repeated we could use use any of colours %0001, %0101, %1001 or %1101 and they'll all come out red – we don't care about the top two bits.

Alternatively, we could be in a pattern assigned to the "sprite" (bottom) palette and want to display yellow. This is colour %10, but it fills the entire palette block from %1000 to %1011, so again we don't care what the lower two bits are, just what the top two bits are.

Using this somewhat redundant palette, we can pack two different four-colour patterns for different parts of the screen into a single sixteen-colour pattern. It's a shame that a halving in bit count ends up quartering the number of colours but I think it's an acceptable solution considering the hardware limitations.

Better drawing, editing, memory and emulation for BBC BASIC on the Sega Master System

Tuesday, 3rd August 2021

I have continued to work on the Sega Master System version of BBC BASIC, and it's feeling much more like a practical version of BASIC than something that was just holding together to run one specific program!

One of the key improvements is to standardise the handling of the different graphics modes. Previously I was using a coordinate system where (0,0) is in the top left corner and all drawing operations were carried out using physical device coordinates (so the screen was treated as being 256 pixels wide and 192 pixels tall). I have now moved the origin (0,0) into the bottom left corner with the Y axis pointing up and scale down all coordinates by 5, effectively making the screen 1280 logical units wide and 960 units tall. This isn't 100% compliant, as the BBC Micro and other versions of BASIC treat the screen as being 1024 units tall, but dividing by 5⅓ is considerably trickier and it would result in the graphics being squished further vertically, so I think using a logical resolution of 1280×960 is an acceptable compromise. I've added some VDU statements to allow you to move the graphics origin as well as define custom graphics and text viewports, so you can control where on the screen graphics and text appear and how they are clipped/scroll.

The output of the "Mandelbaum" program following these changes

I have also changed the default palette to more closely match the one used by the BBC Micro and other versions of BBC BASIC. This isn't too difficult when using the Master System's "Mode 4" as that has a user-definable palette, but the legacy TMS9918A modes have a fixed palette so I've tried to match the default palette as sensibly as I can to the TMS9918A palette. It's possible to change the logical to physical palette mappings under Master System "Mode 4" via a VDU command which writes directly to the CRAM (you can either remap one of the 16 logical entries to one of the stock 16 colours, or supply an RGB value directy to select additional colours) which allows for neat tricks like palette cycling, but the TMS9918A modes currently only let you change the current text/drawing colour, not amend the palette as that's fixed in hardware.

I've also added filled/outlined circle and axis-aligned ellipse PLOT commands using some code written by Darren "qarnos" Cubitt which I originally used in the TI-83 Plus version of BBC BASIC. This code is very capable and fully accepted 16-bit coordinates for its inputs, however it was also originally designed to output to a 96×64 pixel screen so the final plotting was done with 8-bit coordinates ranging from -128..127. Fortunately the Master System's screen also fits in 8-bit coordinates at 256 pixels wide but that's not quite enough information as you also need to be able to tell if a particular point is off-screen (less than zero or greater than 255); simply clipping it against those boundaries will result in a vertical smear on the left or right edge of the screen when drawing outlines. Fortunately I was able to figure out how to modify his code to add some extra clipping status flags to ensure that ellipses were clipped and displayed correctly on any part of the screen.

   
Filled circles and filled/outlined ellipses make these drawings possible

The only graphics operation exposed by the mode-specific drivers before was a simple "set pixel" routine. This is fine for lines but quite slow for filling shapes so graphics mode drivers can now supply a "fill horizontal span" routine for faster shape-filling. If the routine is left unimplemented a stub routine is provided that fills the span using the "set pixel" routine.

I also added a rectangle-filling PLOT command, which is perhaps not the most exciting-sounding graphics operation but it is used to clear the screen so it is at least useful. More interesting is a triangle-filling routine, something I've never enjoyed writing!

Usually I get very bogged down in the idea that the pixels affected when you fill a triangle should exactly fit within the pixels that are outlined by drawing lines between each of the triangle's points, no more and no less. This can be a bit difficult when the triangle is filled by tracing its edges from top to bottom and drawing horizontal spans between them. If the edge is "steep" (its overall height is greater than or equal to its width) then this isn't too bad, as there's only one X coordinate for each Y coordinate where a pixel would have been plotted. However, when the edge is "shallow" (its overall width is greater than its width) there are going to be certain Y coordinates where the line drawn would have had multiple pixels plotted. In that case, where is the boundary of the horizontal span?

The cop-out answer I've used in the past has been to set up three buffers the total height of the screen and to "draw" the three lines first using the same line-drawing algorithm as the line PLOTting command, keeping track of the minimum and maximum X coordinate for each Y coordinate. When it's time to fill the triangle the minimum and maximum X coordinate for each edge can be determined based on the current Y coordinate and a span drawn between them for perfect triangles. On the TI-83 Plus this takes up four bytes per line (minimum and maximum 16-bit values) for a 64 pixel tall screen, with three buffers for the three lines that comes to 4×64×3=768 bytes, pretty bad. On the Sega Master System that would be 4×192×3=2304 bytes, totally unacceptable on a machine with only 8KB total work RAM!

Each face of this 3D sphere is filled by two triangles.

I've instead simply done my best to interpolate from one X coordinate to the other when working my way down the triangle and filling scanlines, doing a bit of extra pre-incrementing and fudging of initial error values depending on whether it's the top half or bottom half of the triangle. My test was to draw the outline of a triangle in bright white and then to fill a dark blue triangle over the top, if any bright white pixels were visible around the outside this indicated a gap. I mostly got it filled, but I then tried my test program on a BBC Micro emulator and found the BBC Micro exhibited similar gaps so I don't think I'm doing too badly! The above screenshot of the 3D sphere was rendered using this triangle-filling code.

I've also been working on improving the line editor. This is called by BASIC when it's asking for a line of text input from you. Previously I'd only implemented adding characters to the end of the line and pressing backspace to remove characters from the end of the line; if you'd typed in a long piece of code and made a mistake at the start you'd need to backspace all the way to the mistake, correct it, then re-type the rest of the line. Now you can use the cursor keys (or home/end) to move around within the line, insert or overwrite new characters (toggled by pressing the insert key on the keyboard) at the cursor position or backspace/delete characters mid-line as required. It sounds like a small thing but it was quite a lot of code to get right and makes a big difference for usability!

Another feature I've added to aid modifying lines of code is the *EDIT command. This takes a line number as an argument and then brings up the program line you've requested in the line editor, ready for editing. The way this works is a bit sneaky, as it's not natively implemented by BBC BASIC! The trick that makes it work is the ability to override two routines, OSLINE (which BASIC calls when it wants to display the line editor) and OSWRCH (which BASIC calls when it wants to output a character).

When a valid *EDIT <line> command is entered, OSLINE is overridden with the first custom routine. This routine doesn't ask for a line of input, but instead copies L.<line> to the input buffer, overrides OSWRCH with a routine that captures data to RAM, overrides OSLINE with a second custom routine, then returns. BASIC therefore thinks you've typed in a LIST statement so it dutifully starts outputting the line via OSWRCH, but this has been overridden to write the characters to RAM rather than the screen. When it's done this BASIC then calls OSLINE again for the next line of input, which brings up the second custom OSLINE handler. This pulls the data from RAM previously captured by the OSWRCH handler, copies it to the OSLINE buffer, and dispays it on-screen as if you'd just typed it in. It then restores the original OSLINE and OSWRCH handlers before jumping back into the stock OSLINE routine, so you can continue editing the line that you'd requested via *EDIT.

All of this hopefully makes entering programs via the keyboard less cumbersome. Of course, not having to type in programs in full every time you wanted to run them would be even better, and an attempt to give BASIC more memory provides another way to load programs in a somewhat roundabout manner.

The photograph above shows the modified Monopoly cartridge that I'm now using to test BBC BASIC on real hardware instead of the modified After Burner cartridge I was using before. The advantage of Monopoly is that it has an additional 8KB RAM on board, which is used to save games in progress. Sega's cartridge mapper allows for on-cartridge RAM to be mapped into the address range $8000..$BFFF, immediately below the main work RAM which is at $C000..$DFFF. If present, then, BASIC's memory range (which runs from PAGE to HIMEM) can be extended by enabling the save RAM and moving PAGE down.

$8000..$BFFF is a 16KB range, though, and I mentioned that Monopoly has an 8KB RAM. This is true, and what it means is that the 8KB cartridge RAM is accessible from $8000..$9FFF but is then repeated ("mirrored") from $A000..$BFFF. The cartridge RAM detection therefore has to check two things: firstly that there is RAM in the first place (which can be verified by modifying memory at $8000 and seeing if those values stick) and how big it is (which can be checked by writing to $8000 and seeing if that has also modified the data at $8000+<RAM size>). Once presence of any RAM has been determined during startup, RAM mirroring is checked in 1KB, 2KB, 4KB and 8KB offsets. If any RAM mirroring is detected, then it's assumed the RAM is the size that was being checked at the time, however if not it's assumed that the RAM is the full 16KB. At this point, PAGE (which is $C000 in a stock machine with no cartridge RAM) is moved backwards by the size of the detected cartridge RAM, e.g. to $A000 for an 8KB RAM and $8000 for a 16KB RAM. This results in 16KB or 24KB total available to BBC BASIC, a considerable upgrade from the plain 8KB work RAM!

An added bonus of this is that when you type in programs they grow upwards in memory from PAGE. As PAGE now starts within your cartridge memory, and that cartridge memory retains its contents courtesy of a backup battery, it means that if your entered program is smaller than the size of your cartridge memory you can restore it when you switch the console back on by typing OLD.

That's very well for life on real hardware, but I continue to do most of my development testing in an emulator. I'm having to use my own emulator as there aren't any other Master System emulators that also include a PS/2 keyboard emulator, but I did end up running into a very weird bug. Certain trigonometric functions in BBC BASIC were producing very wrong values, resulting in very odd-looking output.

These shapes are on the wonk

Even though the Z80 emulator at the heart of the program passed ZEXDOC (an instruction tester that checks documented functionality) I remembered that it had failed some aspect of ZEXALL which checks the undocumented flags too. I re-ran ZEXALL and found the problem was with my implementation of the bit instruction, so I worked on fixing that including emulation of the Z80's internal temporary memptr/WZ register to ensure that bit n,(hl) set bits 3 and 5 of the flag register appropriately. ZEXALL now passes, but unsurprisingly it didn't fix my problem (as I didn't really think that BBC BASIC would rely on undocumented functionality!)

I ended up isolating certain exact values that when plugged into SIN() or COS() would produce incorrect results. I then dug out my old CP/M emulator and tried plugging those values into the generic CP/M version of BBC BASIC (which has none of my code in it!) and that produced the same, incorrect, results confirming that the issue was definitely in my Z80 emulation and not in some flaw of the BBC BASIC port.

After tracing through the code and dumping out debug values at certain points and seeing where it differed to a known good reference emulator I found that the fault occurred in FMUL, and from there I found that at some point the carry flag (which is either rotated or added to registers with RR or ADC instructions) was being lost. RR looked fine but digging into my implementation for the 16-bit ADC I found the culprit: the code is the same for ADD and ADC, but in ADC if the carry flag is set then the second operand is incremented by one first. This produces the correct numeric answer, but if the second operand was $FFFF on entry to the function then adding a carry of 1 to it would cause it to overflow back to 0. As this happens before any of the rest of the calculations are made it means that the final value of the carry flag was calculated based on op1+0 instead of op1+$10000, hence the loss of the carry flag.

Fortunately, fixing this fixed the wonky output demonstrated above and I feel slightly more confident in my emulation! Now I can continue working on BBC BASIC, I've had an idea for a screen mode that has a reduced number of colours but will hopefully let you draw anywhere on the screen without running out of tiles causing odd corruption (as happens in the usual 16-colour mode 4) and without attribute clash (as happens in the TMS9918A Graphics II mode)...

Running BBC BASIC on the Sega Master System

Thursday, 22nd July 2021

I've recently been spending some time finding a way to run BBC BASIC on the Sega Master System, inspired by BASIC Month 6: The Mandelbaum Set on the RetroBattlestations Reddit community.

When this month's program was first announced I tried running it on my only unarguably retrobattlestation, my Cambridge Z88, but the screen's low (vertical) resolution didn't do the program much justice.

I thought this gave me two options:

1. Control some external piece of retro tech to produce higher-resolution output (e.g. a printer or a plotter.
2. Port a BASIC interpreter to another retro system with a higher resolution display and run the program on that.

Unfortunately, I don't own any old printers or plotters (or a Logo-like turtle!) so option 2 seemed my best option. I had some experience adapting Richard Russell's BBC BASIC (Z80) to run on the TI-83 Plus calculator so I thought I should pick the Sega Master System as that also has a Z80 CPU in it. Here are some rough specs:

• 3.58(ish)MHz Z80 CPU.
• 8KB work RAM.
• TMS9918A-derived VDP for video with 16KB dedicated VRAM accessed via I/O port.
• SN76489-derived PSG for sound.
• Two controller ports with six input pins and two pins that could be configured as inputs or outputs.
• Software loaded from ROM cartridge or card slot with a small BIOS ROM that detects whether a cartridge or card is inserted (and if not, runs its own built-in game).

There is some precedence to this endeavour with the computer version of Sega's SG-1000, the SC-3000, which came with a keyboard and had BASIC ROM cartridges.

I wanted to try to keep this project as retro as possible, so no modern microcontrollers as I've been accused of cheating by using them in the past. I did have to make a couple of adaptors to allow me to plug in a keyboard and to give the Master System a serial port to load or save programs over – more about these later!

Loading BASIC onto the Master System

The first problem was getting BASIC onto the Master System at all. As my Master System doesn't have BASIC in ROM (it comes with Hang-On, which is perhaps more fun but less likely to handle the Mandelbaum set) I'd need to load the program onto a cartridge or card. Master System cartridges usually contain a mapper circuit to handle bank switching in the lower 48KB of the Z80's address space (the upper 16KB contains the 8KB work RAM, appearing twice) which is normally integrated directly into the ROM chip for the game, however there are a handful of games that have separate mapper chips and ROM chips and the ROM chips that Sega used have a pinout that is extremely close to that used by common EEPROMs (e.g. 29F010,
49F040) with only a couple of pins needing to be swapped around. After Burner is one such cartridge, so I've modified an old copy to let me plug in flash memory chips that I can program with the version of BASIC I'm working on. A switch at the top of the cartridge lets me switch back to the original pin configuration if I want to play the original copy of After Burner.

Typing commands into BASIC

With a flashable cartridge to hand I was able to assemble a version of Richard Russell's BBC BASIC (Z80) with some simple code stubs in place to direct text output to the screen. Output is useful but only half the story, we still need to be send commands to BASIC!

The Master System doesn't have its own keyboard, so I'd need to find some way to interface one to it. I have previously used PS/2 keyboards in a number of projects, as they are pretty simple to deal with. Electrically, they use two open collector I/O lines for bidirectional data transfer (one as clock, one as data), and fortunately each controller port of the Master System has two pins that can be configured as outputs which can therefore be used to interface with a keyboard (either left as inputs and pulled weakly high in their idle state, or driven low as outputs for their active state).

It acts as a pass-through cable so you can still have a regular controller plugged in when using the keyboard.

The two pins on the Master System control port that can act as outputs are TH and TR; TH is normally used by light guns to latch the horizontal counter in the video chip so is unused with normal controllers so it's no great loss here, but TR is the right action button (marked "2") so by using this pass-through adaptor you do unfortunately lose one of the controller buttons. However, you do gain a hundred or so keyboard keys, so I don't think it's too bad a compromise...

For the software side of things I adapted my Emerson AT device library, previously written for the TI-83 Plus calculators, to the Sega Master System hardware. This library handles the low-level AT device protocol and also translates the raw keyboard scancode values to the corresponding characters.

Saving and loading programs

At this point I was able to type in BASIC programs and run them on the Master System, which was pretty neat! However, I was still working on adding new features (e.g. drawing commands for graphics) and having to type in the entire Mandelbaum program after every change was going to get pretty exhausting. I could bake it into the ROM but that seemed like cheating, so I thought I should try to find a way to load the file from an external source. A floppy disk or tape cassette would seem authentically retro but adding a floppy drive controller to the Master System would be a fairly complicated task and I don't have a suitable data cassette recorder to even attempt loading from tape so I thought some sort of file store accessible over a serial port would be a good option.

The Z88 has a serial port and can act as a remotely-controlled file store when running the PC Link software (with the protocol documented here. This seemed like a good choice, if not for the fact that the Sega Master System doesn't have a serial port of its own. To get around this, I added one, using a MAX232 chip to adapt the Master System's 5V logic levels to RS-232 compatible ones so I could plug in a null modem cable from the Z88 or my PC without accidentally frying the Master System with -12V.

I wrote some code that bit-bangs the serial data over the controller port lines using the timing loop code I wrote for a previous BASIC Month (Crisps Tunes) to support rates between 19200 and 300 baud. 19200 baud is somewhat unreliable but that's OK because the Z88's 19200 baud is unreliable too, so the default 9600 baud speed does a good job. RTS/CTS handshaking has to be implemented, as there is no hardware serial support on the Master System and it needs to be actively polling the port to receive any data. In doing so I noticed one awkward fact about my PC's serial port - if you de-assert RTS it will continue sending data until its buffer is empty, presumably only checking the RTS line when it's about to top up the buffer. In practice this means that even if I change RTS virtually as soon as the start bit for the first byte is received, the PC will continue to send up to 256 bytes before stopping. To get around this I added a serial receive buffer that immediately checks for the next byte even after asserting RTS, and this seems to have done the trick.

The protocol used by PC Link requires acknowledgement after every single byte so is very slow but at least it's reliable. I plumbed the PC Link code into BASIC's LOAD and SAVE which makes loading and saving programs as transparent and easy as if you had a floppy disk in the system instead!

Pressed for size

The Master System has 8KB of RAM. Of this, 16 bytes are mapped to special hardware functions and BBC BASIC reserves 768 bytes for itself, so we're already down to 7,408 bytes. I initially reserved 256 bytes for my own needs (display settings, VDU command buffer, serial port status, keyboard status etc) bringing it down to 7,152 bytes. The 16KB of display memory is not directly accessible to the CPU, so it can't be used for additional work RAM, and having to access it indirectly via I/O ports is very slow but I can't afford to mirror parts of it in RAM for speed.

Initially the Mandelbaum program ran well enough by stripping out comments, but I then added sound support (with eight 13-byte envelope definitions, and four channels with their own state, copy of their active envelope and a command queue for 32 bytes per channel, adding around another 140 bytes of memory usage) and the program stopped running with a "No room" error during execution (performing a square root operation, of all things!) so I guess the tolerances were very tight. I went and combine more lines of code into single lines and replaced two-letter variable names with single-letter ones (no, really) and it was able to run again but I don't think 8KB is a particularly comfortable amount of RAM for a BASIC computer!

Further design considerations

The version of the BASIC host interface used here is very much a work-in-progress. I would need to extend it considerably to be useful, including:

• Fuller support of different VDU commands, e.g. redefining character shapes, changing the text and graphics viewports, better colour handling (differentiating between logical palettes and physical palettes).
• Better support of other modes (so far only TMS9918A "Text" and "Graphics II" modes are used, there is a Master System-specific "mode 4" but that lacks graphics support and only takes advantage of hardware scrolling for extremely fast program LISTing).
• Implementation of more graphics commands - so far only PLOT 4 (MOVE) and PLOT 5 (DRAW) are implemented. They also use a non-standard coordinate system with (0,0) in the top left of the screen and a screen resolution of 256x192, whereas for standardisation with other BBC BASIC implementations this should move (0,0) to the bottom left and use a logical resolution of 1280x1024 or similar.
• Support of other file systems rather than rely on a Z88 running PC Link, e.g. using I²C EEPROMs (as they use two open collector pins, so could be plugged into a controller port via a passive adaptor).
• Support of extra RAM, either integrated directly on a custom cartridge or using the battery-backed SRAM supported by some other cartridge types.
• Native controller support via BASIC ADVAL command (at the moment you can access the controller ports directly with GET(&DC))

This is before even getting into adding anything machine-specific (e.g. to take advantage of scrolling tilemaps or hardware sprites), but getting the BASICs down (and consistent!) is a very important starting point. Consistency is useful, after all I was able to get the original program running under BBC BASIC with very minimal changes to it.

But, in the short term, I have at least succeeded in what I set out to do, which was to run the Mandelbaum program on a retro "computer" wholly unsuited to it!

The above video provides another demonstration of the setup – playing the Cold Tea music demo, albeit with a heavily stripped-down version of the visuals as the Master System doesn't have anything that can match the capabilities of the BBC Micro's teletext mode 7.

Migrating SVN repositories to GitHub, maintaining history even on addition/removal of a trunk folder

Friday, 16th July 2021

Years ago I had a few projects hosted in Google Code's SVN repositories, but Google closed Code down in 2015 and though I backed the files up locally via svnsync to carry on working on the projects the code was never made publicly available again.

I decided it would be a good idea to move these repositories to GitHub, but quickly ran into the problem that during the initial migration from my local repository to Google Code's repository I had been forced to move all of the code into a trunk folder (I do not generally use SVN's branching or tagging features) and attempting to synchronise from SVN to Git lost all revision history prior to that change.

After scouring the Internet and many dead ends I came up with the below solution which has worked for me. As far as I can tell the Git tools used are designed for regular synchronisation of repositories and not a one-shot bulk migration. There are separate tools that claim to do a better job with less effort but getting them up and running on my Windows system probably takes longer than figuring out how to do it with Git's own tools...

First of all, you'll need to create a user list that maps your SVN user name(s) (left) to your name and email address (right) used on GitHub, like this:

Ben = Ben Ryves <benryves@benryves.com>
Benjamin = Ben Ryves <benryves@benryves.com>
benryves = Ben Ryves <benryves@benryves.com>
benryves@benryves.com = Ben Ryves <benryves@benryves.com>

git svn clone --no-metadata --authors-file=users.txt -r 1:99 file:///D/SVN/Google/brass-assembler temp

• There is no --stdlayout parameter. This is because this makes the assumption that the repository follows the conventional trunk/tag/branch folder used in some SVN repositories, but this repository doesn't make use of such a structure (at least until revision 100!)
• The -r 1:99 argument limits the operation to be between the revisions 1 and 99. This is when all the files were in the root of the repository, before they were moved in revision 100.
• The SVN path looks like it's missing a colon, but this is intentional – at least on Windows the Git tools fail if the path contains a colon like it would when using the SVN tools for local paths, so remove that colon.

This may take a while to get started but eventually it should start synchronising the SVN repository to the new Git one in the temp folder up to (and including) the last revision where everything was in the root (99).

Once that has happened, you can perform the trick that makes this work. Open the file temp\.git\config in a text editor and in the [svn-remote "svn"] section you should find the line fetch = :refs/remotes/git-svn. As revisions after this point were moved into a trunk directory, we need to change this to fetch from trunk instead, so change this line to fetch = trunk:refs/remotes/git-svn:

[svn-remote "svn"]
	noMetadata = 1
	url = file:///D/SVN/Google/brass-assembler
	fetch = trunk:refs/remotes/git-svn
[svn]
	authorsfile = C:/Users/Ben/users.txt

cd temp
git svn fetch -r 101:HEAD
git merge remotes/git-svn

cd ..
git clone temp Brass3
rmdir /s /q temp

cd Brass3
git remote rm origin
git remote add origin https://github.com/benryves/Brass3.git
git push origin master

One thing I realised during all this was that Brass 3 had some problems running on 64-bit versions of Windows – the help application is completely non-functional, for example, and crashes silently to desktop. I dug into the code to fix it, only to find out that I'd already done so two years ago, and even got as far as rebuilding the installer package but then just forgot to upload it to the Internet. So, I'm very sorry for the delay, but I have now uploaded "Beta 14" to the Brass 3 page.

Making your own Dreamcast MIDI Interface Cable

Saturday, 1st May 2021

I love an unusual accessory for a video game console or computer, and one such accessory is the Dreamcast MIDI Interface Cable, HKT-9200, which allows you to connect MIDI devices to the Dreamcast console's serial port. Only released in Japan and with only one piece of software released for it — the O･to･i･re (お･と･い･れ) sequencer – these are a somewhat hard to find accessory nowadays and prices for second-hand units are far beyond what I could hope to afford (at the time of writing there are two on eBay, both for over £300).

Fortunately, the user darcagn on the Obscure Gamers forum took some photos of the insides of the interface box and from that I could make a pretty good guess as to how the cable works.

MIDI uses a serial protocol running at 31.25Kbps (a speed that can be easily derived by dividing a 1MHz clock by 32). Rather than signal "0" or "1" bits with different voltage levels (as with a PC's RS-232 serial ports, for example, which commonly uses +12V for a "0" and -12V for a "1") it uses a current loop, with 5mA current on for a "0" and current switched off for a "1". To avoid ground loops, which are a big concern when working with audio as they can introduce intereference (e.g. a mains hum) on recordings, the two connected devices are electrically isolated with an optoisolator in the receiver.

At the very least I therefore expected to see some sort of optoisolator circuit on the adaptor's MIDI IN port and some sort of output buffer circuit on the adaptor's MIDI OUT port to convert between MIDI's current loop signalling and the Dreamcast's 3.3V logic on its serial I/O pins, and that is indeed what you can see from darcagn's photos. My worry was that there might be some additional Dreamcast-specific hardware inside the box, but fortunately there isn't – it's all off-the-shelf parts. My main concern was how everything was connected, as this can't be completely seen from the photos: would sending MIDI IN data to the console's serial port RX and relaying data from the console's serial port TX to MIDI OUT be enough? Some experimentation would be necessary!

Building a serial port connector

As mentioned above, the MIDI interface cable's box doesn't contain anything Dreamcast-specific, however this box is connected to the Dreamcast's serial port using a proprietary connector. To try anything out I'd need to find a way to connect a circuit to this port:

Fortunately, the port's contact pitch is the same as a PCI Express slot, and I was able to find a PCI Express slot for £5 which could be used to make multiple connectors! It will need to be cut down to size (and in half, as the Dreamcast's serial port only has contacts on one side rather than both sides of a PCI Express card) but with a bit of work will do the job.

The above photos show the process of cutting the PCI Express slot down to size. The Dreamcast serial port has 10 connectors in it, so a block is cut that is 12 connectors long using a cutting disc – as this is quite a rough process an extra sacrificial connector is left on each end as it doesn't matter if this gets mangled by the cutter. The block is then cut in half, leaving more of the support structure from the bottom of the slot on the side of the slot we're going to be using. The outer two connectors are then removed if they haven't already been damaged, leaving the central 10 connectors, and the outer plastic is brought to the final width and tidied up with some hand files. The fit of the connector should be tested against the Dreamcast's serial port:

There shouldn't be too much side-to-side movement but the connector will be very loose without something to hold it down against the contacts. In my case I found some 2mm thick ABS plastic sheet was the perfect material to make the backing piece for the connector, though you may find your choice of material depends on the thickness of your PCI Express slot. It will need to be 13.5mm wide (about ½") and a decent enough length to fit inside the enclosure you're going to use for the plug – in my case 4cms was about right. The plastic can be cut by scoring it with a knife and then snapping it over the edge of a table.

You should also drill some shallow holes in the plastic, with the centres of the holes being 3mm from the end and 3mm from the sides. The serial port has a couple of bumps stamped into the metal surround of the serial port and these matching holes in the plastic piece allow it to snap into place. A stripboard track-cutting drill is perfect for this task!

In my case the PCI Express slot also has a slight lip that prevents it from sitting flush against the backing support. I could have filed this flat but the connector is quite fragile so I didn't want to risk damaging it so I ended up filing a corresponding channel into the bottom of the backing support piece.

At this point make sure that everything fits. If it does you can start wiring up! The photos below show the process – heat-shrink tubing and strain reliefs are very useful, so don't forget to install them before soldering!

Putting a scrap piece of circuit board material between the two rows of pins also makes soldering much easier. In my case I only soldered the six pins required for this MIDI interface cable:

• 1: +5V
• 3: GND
• 4: RX
• 5: TX
• 8: GND
• 10: +3.3V

Pins are numbered from left to right when looking at the serial port at the rear of the console (if in doubt, you can check the voltages of the end pins against the console's metal chassis ground).

At this point you may wish to double check that your solder connections are made correctly and that nothing is shorted out – try with the cable plugged into the Dreamcast as well, and check that adjacent pins are not shorted together (the only two that should be shorted are pins 3 and 8, the two GNDs). If you're happy with that you can glue the connector onto the backing:

You may notice that this is actually a different connector in the photo to the previous one, and that's because I accidentally got glue into the connector's springy contacts and jammed them so had to start again – definitely not a fun mistake to make, so be careful!

Fortunately, the second one went more successfully. The glued connector snaps in and out of the console with a nice reassuring click thanks to the two holes drilled into the top surface. Before switching on the Dreamcast I tried wiggling the cable around to ensure that even when treated roughly it woudn't short out adjacent pins. When I was happy this was the case I switched on the Dreamcast and ensured that I was getting a consistent +5V from pin 1 and +3.3V from pin 10 – as these pins are at the far end of the connector these are the ones that are more likely to have problems with crooked connectors. In my case I found there was an intermittent fault with pin 1's +5V. This was because I hadn't glued the connector on particularly straight and so pin 1's connector was slightly back from the edge of the backing piece. I very carefully filed the backing piece's edge so that it was flush with the slightly wonky PCI Express connector, after which pin 1 made reliable contact.

When you're completely happy with the connector, you can make it more robust by putting it inside an enclosure. I have some very small project boxes that are perfect for this sort of thing, it's a bit bulky when compared to the official Sega product but it does its job well here and doesn't bump into the power connector or AV port connector.
I cut a slot in one end of the box for the connector to stick out of and a notch in the other for the cable strain relief to clip into. I surrounded the console's serial port with a few of layers of masking tape to ensure that when the connector was inserted there was still a small gap between the case and the plug to make sure that it could always be fully inserted and not held back by interference from the case (it also protected the console shell from accidental strings of hot glue!) I then plugged the connector into the Dreamcast, made sure that everthing was neatly lined up, and secured the parts in place with copious amounts of hot glue. Once this had set I added more hot glue to the rear of the connector to make sure it was all held as securely as possible, and then screwed the enclosure shut.

With all the effort spent on the Dreamcast end of the cable, don't forget about the MIDI interface box end! I'm fond of JST-XH connectors so crimped one onto the end of the cable, ready to plug into the circuit board. The finished cable is seen above!

Building a prototype MIDI interface cable

With a Dreamcast serial port cable to hand I was able to experiment and see what happened when using the O･to･i･re (お･と･い･れ) sequencer. Here's the circuit I ended up building on a breadboard:

On the very left is the first serial cable connector I tried constructing. It looks more yellow than the final version as I used a piece of pad board for the backing piece instead of the 2mm ABS plastic – I thought I was going to solder the PCI Express connector piece to a circuit board rather than directly to the cable's wires, this turned out to be a mistake. One advantage is that it does have eight wires soldered to it – I thought I'd see what RTS and CTS were doing, but I ended up not using them. The MIDI port to the left is for MIDI OUT and has a large black chip to act as a buffer, the port to the right is for MIDI IN and has a small white chip as an optoisolator. Here's the corresponding circuit diagram:

The position of the ports is flipped in this diagram when compared to photo of the breadboard, but it otherwise matches up!

On the left is the MIDI IN port. This uses an H11L1 optoisolator as they are still reasonably easy to get hold of today, are fast enough for use with MIDI and can be run directly from +3.3V. Its output on pin 4 is open collector so it needs the 270Ω pull-up resistor to the +3.3V rail. A +5V-demanding optoisolator could also have been used if it had an open collector output and the pull-up resistor on its output was still tied to +3.3V (we don't want to run +5V into the Dreamcast's +3.3V logic!) but this makes the wiring a little more complicated so sticking to a +3.3V-compatible part makes life easier.

The 36Ω resistor between the output of the MIDI IN circuit and the serial port's RX pin is there because it is in the official cable. The official cable also places ferrite beads on every I/O pin (and has a ferrite bead clipped onto the cable itself) which I have not replicated in my own cable, but they can't hurt and I suppose it could protect the console from certain direct shorts!

The MIDI OUT port uses a 74HC365N to convert from the serial port's +3.3V logic to +5V to drive the MIDI output. MIDI signals can be run from +3.3V (it's the flow of current that's more important than the selected voltage) but +5V seems more typical so I thought I'd stick with that, and as we need to buffer the signal anyway (I'm not sure how much current the Dreamcast's serial ports are designed to sink or source) using the +5V supply we have available to us made sense. The voltage threshold for a "high" input signal is probably a bit too high with the 74HC365N – +3.3V is pretty close to the recommended values in the datasheet, so the 74HCT365N version of the chip would give you more margin for error and would be a drop-in replacement. In my testing the 74HC365N does work well, though, and it's what I had available.

The cable used in this prototype has eight connections rather than the six in the final. This is because I did experiment with the RTS (pin 6) and CTS (pin 7) signals from the console, however as far as I can see these are just pulled low and high respectively and do not change from the moment the console boots, even when sending and receiving MIDI data within the O･to･i･re (お･と･い･れ) software. If they were planned to be used for some purpose then I'm not sure what, and with only one piece of software released to test with I'm not sure I'll find out.

In any case, with this circuit data from the MIDI IN port is translated to +3.3V logic levels suitable for the Dreamcast's serial port input (RX) and translated back from the serial port output (TX) to a current loop suitable to drive a device connected to the MIDI OUT port. In practice it seems this MIDI OUT port acts more like a MIDI THRU with the O･to･i･re (お･と･い･れ) sequencer – any data sent to the MIDI IN port comes straight back out of the MIDI OUT port, however if you record some notes in the sequencer and play them back afterwards they aren't played back out of MIDI OUT. At first I wondered if I'd made a wiring error (accidentally connecting MIDI IN straight to MIDI OUT) but the MIDI OUT is indeed under control of the software as it will stop relaying messages on certain screens.

Putting it all together in a nice box

I ended up using my usual pad board construction technique to build the final device:

I followed the circuit diagram I'd drawn earlier (rather than copy the breadboard circuit) to ensure the diagram was correct. This was all assembled to fit in a ready-made ABS enclosure, into which I cut some holes. Historically I've had a hard time cutting neat round holes in plastic enclosures – I have a few hole saws and these are great for cutting through wood or acrylic but when trying to cut ABS they tend to bind and either rip the box out of the vice or just shatter it. I normally resort to drilling lots of very small holes around the perimeter of the circle and then try to file it to size, which is OK for buttons or sockets with overhanging parts to hide the inaccuracies but wouldn't do here! For this project I bought a very cheap step drill set on eBay (three bits for a fiver) with zero expectations but it did an excellent job, it kept the centre hole I'd started from and didn't need very much cleaning up. I wish I'd bought one sooner!

The other enclosure challenge to deal with was labelling the two ports. I normally get away without labelling my projects because the function of each port can be guessed quite easily (e.g. inputs and outputs normally only plug in one way, or player 1 is on the left and player 2 is on the right) but in this case there's not much convention for where MIDI OUT and MIDI IN go (though the original Dreamcast MIDI interface cable puts MIDI OUT on the left, as does my M-Audio Midisport). For this I thought I'd try using some dry transfer lettering designed for model-making, and it seems to do a great job!

I'm pretty happy with the final outcome – it's a bit of a pain making the serial cable for connection to the Dreamcast, but the end result works well and it saves spending an absolute fortune on the original Sega accessory.

Now I can get on with making music with my Dreamcast!