-- Leo's gemini proxy

-- Connecting to gemini.exoticsilicon.com:1965...

-- Connected

-- Sending request

-- Meta line: 20 text/gemini; charset=utf-8

Candlelit console


Introduction


Thirty years ago, you might have had a `colour temperature' knob on the front of your brand new high-end Super VGA monitor.(This would likely have been at work, of course, as most home users could barely afford 16-colour EGA.)


Twiddling this control towards `warm' during those late nights at the office probably felt good on the eyes, whereas during the day the crisp, accurate color rendition of `cold' was more productive.


> A quick side note: in some contexts, the term `night mode' is used just to mean a light-on-dark colour scheme, which is absolutely not what we're talking about here. In the true spirit of 21st century recycling, it seems that even this name seems to have been recycled to refer to more than one concept!


In recent years, this idea has been picked up, recycled, and presented to the consumer as something new. It's no longer marketed as `colour temperature', it's now usually, `night mode', or something along those lines. Just about every new electronic device with a screen now has it, and plenty of desktop environments have introduced it too.


Of course, the one desktop environment of note that has been sorely lacking any kind of software-based colour temperature correction is the framebuffer console in OpenBSD.


Until now, that is...


In today's presentation


We could just present this all neatly wrapped up as a short kernel diff and some usage instructions...


Instead, today we'll take a look at and de-mystify the actual process of coding all of this new functionality.


So if you're a seasoned C programmer who just isn't yet very familiar with the kernel internals, now is the perfect time to learn:


How to add a new sysctl knob to the kernel and parse it's value.

How to add a new escape sequence to the terminal emulation code.

How to adjust the device colour pallette on the fly.


Wow, all that and a bag of chips sounds too good to be true, huh? Well, if you like this content from Exotic Silicon, and want to see more then please show your appreciation by linking to us from reputable websites, and mentioning us on social media.


A set of patches to implement the new functionality can be found near the bottom of this page, for any impatient readers who just want to see the end results.


First steps - testing a new pallette


Our first goal to is create a new colour pallette with a warmer hue, by reducing the blue content. If you've read our `reckless guide', you'll already know that the colour pallette is defined in sys/dev/rasops/rasops.c. If we just wanted an alternative pallette all the time, we could simply change the RGB values defined there. However, we want a choice of at least two pallettes, the normal one and a warmer one.


Read our 'Reckless guide to OpenBSD' here if you haven't already!


At this point comes our first design decision. Should we manually define an additional set of sixteen warm colours, and then simply add sixteen to the index of the chosen colour when we want to reach into our second pallette? Or should the alternative colours be somehow derived mathematically on the fly from the original pallette?


Both approaches would be reasonable, especially as there are a large number of unused entries in the rasops_cmap array that holds the RGB values. However, it seems more straightforward and practical to generate the new pallette automatically, so we'll use this method here.


Unsurprisingly, there isn't really much experimentation to be done in order to find a suitable algorithm for reducing blue content. Simply shifting the 8-bit blue value right by one bit will reduce the intensity of the blue channel, and in practice this does indeed produce a visually pleasing effect.


We can test this by adding a few lines to sys/dev/rasops/rasops32.c, (assuming, of course, that we are using a 32-bit non-byteswapped display, which most users of modern desktops or laptops will be).


In this case, the variables f and b will contain the 24-bit RGB values for the foreground and background colours respectively. All we need to do is to shift the lower eight bits right by one bit.


 /* Reduce blue component */
 int blue;
 blue=(b & 0xff);
 blue=(blue >> 1);
 b=(b & 0xffff00) | blue;
 blue=(f & 0xff);
 blue=(blue >> 1);
 f=(f & 0xffff00) | blue;

This logic can be expressed more compactly in C as:


 b=(b & 0xffff00) | ((b & 0xff)>>1);
 f=(f & 0xffff00) | ((f & 0xff)>>1);

If you do any experiments of your own within the rasops code, be careful about adding printf calls for debugging purposes. This will often panic the kernel, as certain code paths then become called recursively.


After re-compiling the kernel and re-booting, we're greeted with the effect that we were hoping for.


Assembly optimisation interlude for curious readers


The bit shifting calculation above is actually much easier in X86 assembler than it is in C.


This might sound surprising, but as any elite hacker with a firm grasp on X86 assembler should know, we can just do:


 asm volatile ("shrb %%al;" : "+a" (b) : :);
 asm volatile ("shrb %%al;" : "+a" (f) : :);

For those readers who are not familiar with X86 assembler, the way this works is that we load the entire 32-bit value into a 32-bit register, %eax, but then perform the right shift only on the 8-bit register that corresponds with the lower eight bits of %eax, which is %al. The final 32-bit value is then read out from %eax again.


This is typically even more efficient than the code produced by an optimising compiler.

To demonstrate this point, consider the following short C program:


 int main()
 {
         int rgb=0x00ffffff;
         rgb=(rgb & 0xffff00) | ((rgb & 0xff)>>1);
 }

Compiled with clang version 11.1.0 on OpenBSD 7.0-release, and disabling optimisation by using -O0, we get the following assembly output for the bit operations:


 movl      $16777215, -4(%rbp)     # Store 0xFFFFFF in `rgb'
 movl      -4(%rbp), %ecx          # Copy `rgb' to ecx
 andl      $16776960, %ecx         # ecx & 0xffff00
 movl      -4(%rbp), %edx          # Copy `rgb' to edx
 andl      $255, %edx              # edx & 0xff
 sarl      $1, %edx                # edx >> 1
 orl       %edx, %ecx              # ecx = ecx | edx
 movl      %ecx, -4(%rbp)          # `rgb' = ecx

With optimisation enabled, things are slightly more complicated. Since our computed value is not actually used, the compiler will simply optimise it away completely. Even if we make use of the value by adding a final call to printf, the compiler will still optimise away the actual calculation, because all of the arguments are constants, so the result can be computed at compile time as 0xFFFF7F.


Changing our code to call arc4random_uniform to set the initial value of variable rgb, forces the compiler to produce code to do the bit shifting at runtime. Compiled with optimisation level -O3, we get this:


 movl      %eax, %ecx              # Copy the value returned from arc4random_uniform from eax to ecx
 andl      $16776960, %ecx         # ecx & 0xffff00
 shrl      %eax                    # eax >> 1
 andl      $127, %eax              # eax & 0x7f
 leal      (%rax,%rcx), %esi       # esi = (rax + rcx)


Huh? What's leal?


Note the use of the lea opcode, `Load Effective Address', to perform the logical `or' operation. This might be un-intuitive for readers with a background in assembly coding on non-X86 platforms. The lea set of opcodes, of which leal is one, is intended for calculating addresses by doing addition and bit shifting. But since the lea family of opcodes doesn't actually load anything into the calculated address, we can use leal as a convenient general purpose addition and bit-shifting instruction. An optimising compiler typically will use the lea instruction for these sorts of operations.



However, with our hand-grafted in-line assembly, the compiler produces the following:


 movl      $16777215, %eax         # Store 0xFFFFFF in eax
 shrb      %al                     # eax=(eax & 0xffff00) | ((eax & 0xff)>>1);

Which is clearly much better.


Fun observation


It's worth noting that despite this code being run for every character painted to the screen, the performance hit is almost un-measurable, especially with the assembler version.


A quick colour test chart


Since a large amount of the output on the console is by default white text on a black background, if we're going to be making adjustments to the colour rendering it would be useful to have a program that outputs some sort of colour test chart.


Although userland programs should usually consult the terminfo database to obtain suitable escape sequences for displaying colour on the current output device, in this case we can happily put the appropriate escape sequences directly into the code, as this program is specifically intended to display the colour test chart on the OpenBSD framebuffer console, (in it's vt100 emulation mode). This also ensures that the program will produce the correct output even if the wrong terminal type is selected.


These are the escape sequences we'll be using:


 [ESC][0m       Reset all attributes to default values.
 [ESC][1m       Set bold mode, (displayed as high intensity).
 [ESC][3Xm      Set foreground to colour X, (ranging from 0-7).
 [ESC][4Xm      Set background to colour X, (ranging from 0-7).

Note that in this article, ESCAPE, 0x1b, is represented with the following sequence: [ESC]


We can create the colour chart quite easily with the following short shell script:


 #!/bin/ksh
 for i in 0 1 ; do
 echo -n [ESC]["$i"m
 for b in 0 1 2 3 4 5 6 7 ; do
 for f in 0 1 2 3 4 5 6 7 ; do
 echo -n [ESC][3"$f"m[ESC][4"$b"m' TEST '
 done
 echo "[ESC][49m[ESC][39m"
 done
 done

Be aware that in most shells, to enter the escape character represented here by [ESC], it's necessary to enter the `literal' control character first by typing control-v, followed by the escape key. This will be displayed on the console as ^[. The above escapes sequences all immediately follow this with a further regular [ character.


This chart can be used not only to evaluate the visual esthetics of an alternative colour pallette, but also to check that different combinations of foreground and background colours remain visibly distinct.


Moving the code to a better place


Before implementing a new sysctl to allow us to activate and deactivate the alternative colour pallette whenever we want to, we should consider whether there is a better place to put our code than in the middle of the rasops32_putchar function. This was convenient for a quick test, but the extra code is running every time a character is drawn, so we are performing the exact same calculation repeatedly. Although performance is very good with this specific single bit shift of the blue value, if we later decide to expand our code to include other effects then the extra CPU load might become noticeable.


Also, if we wanted to support other colour depths we would need to patch the corresponding functions in each of rasops24.c, rasops15.c, and so on.


The various bit-depth specific functions work with a device colour map, rather than the raw RGB values which are defined at the beginning of rasops.c. The device colour map is computed once during initialisation of the display, in the function rasops_init_devcmap, and then stored for re-use.


This means that we could insert our colour-modifying code in the rasops_init_devcmap function, and then call it to re-calculate the device colour map whenever we change the pallette via our new sysctl. Doing this would completely eliminate the per-character overhead.


Device colour maps


Whereas the colour map values in rasops.c are defined as 24-bit RGB values, in the device colour map colour values are stored in the format that the hardware expects them to be in. This greatly simplifies and speeds up the process of plotting each pixel that is required for every character drawn.


In the case of 32-bit colour, the transformation is usually straightforward, as the only difference between the raw 24-bit values and the device colour map is an extra byte of padding to align the data for each pixel to a 32-bit boundry:


 32-bit RGB
 0x00000000RRRRRRRRGGGGGGGGBBBBBBBB

A 15-bit display would usually expect the data in this format:


 15-bit RGB
 0x0RRRRRGGGGGBBBBB

Byte-swapped versions of these formats also exist, but hardware using them is less common:


 Byte-swapped 32-bit:
 0xBBBBBBBBGGGGGGGGRRRRRRRR00000000

 Byte-swapped 15-bit:
 0xGGGBBBBB0RRRRRGG

The important point to note is that the bit-depth specific functions such as rasops32_putchar, only deal with the device colour map values, and not the raw 24-bit RGB values in rasops_cmap[].


Looking at the rasops_init_devcmap function, we can see that bit depths of 1, 4, and 8 bits per pixel use a fixed hardware pallette which is not based on the RGB values in rasops_cmap at all. This means that our colour changing code will have to be limited to bit depths of 15, 16, 24 and 32 bpp.


The single argument supplied to rasops_init_devcmap is simply a pointer to a rasops_info structure. This is effectively an opaque cookie to access the parameters of the display in question. We can ignore the first part of the function which deals with bit depths, (as given by ri->ri_depth), of 1, 4, and 8. The next section of code, which creates the device colour map for the remaining bit depths, is less complicated than it initially appears.


Variable p is initialised to point to the first byte of rasops_cmap, and will iterate over the red, green, and blue values for each defined colour, in other words, each entry in the table that was defined at the beginning of the file. We loop through the 16 defined colours with variable i, and the code within that loop actually does the job of packing the bits into variable c in the format required by the hardware. The values of ri_Xnum and ri_Xpos, where X is either r, g, or b, determine the number of bits for that channel, as well as their position within the final value.


If we look at rasops.h, we can see a comment saying that if ri_Xnum and ri_Xpos are set to zero, then default values will be applied. Looking in rasops32.c, at the very beginning of rasops32_init, we can see the code that actually does this. In fact, it only checks ri_rnum, and proceeds to set all six variables to default values if ri_rnum is zero.


However, looking very carefully, we can see that these default values actually set the positions of the red and blue values within the final 32-bit value to be swapped. This seems unusual, as it would produce entries in the device colour map in the following format:


 32-bit BGR
 0x00000000BBBBBBBBGGGGGGGGRRRRRRRR

This BGR ordering is the opposite of the order set up by the graphics drivers in /usr/src/sys/dev/, as can be seen by running the following simple grep command:


 # grep -r ri_.pos /usr/src/sys/dev/* /usr/src/sys/dev/rasops/rasops??.c

Since all of the graphics drivers explicitly set values for these variables, the default values will not be used, so this point is somewhat moot. It does seem interesting, though, and might even be a bug.


Adding our colour pallette changing logic to rasops_init_devcmap


If we wanted to support all combinations of 15, 16, 24, and 32 bpp hardware, as well as byte-swapped variations, then we would need to add our code to the assignments made to variable c at the beginning of the loop, for each of the red, green, and blue channels.


This is certainly possible, but if we restrict our support to 24 and 32 bpp modes, and ignore byte-swapping, our changes to the existing source code can be far simpler and less intrusive, which is useful if we intend to port our local changes to future versions of the OpenBSD kernel every six months.


To test the concept, we simply need to insert either one of the two versions of the line of bit shifting code immediately before the final assignment of c to the entry in ri_devcmap right at the end of function rasops_init_devcmap.


 X86 assembler version:
 asm volatile ("shrb %%al;" : "+a" (c) : :);
 Generic C version:
 c=(c & 0xffff00) | ((c & 0xff)>>1);

So we are modifying the value of variable c, which is already in the format that the hardware is expecting, just before assigning it's value to the entry in the device colour map array.


If we now also remove our previous changes to rasops32.c, re-compile and re-install the kernel, then reboot into the newly compiled kernel, we will once again see our yellow-tinted output on the console.


The difference is that our code is now being run just once, at the time of display initialisation, rather than every time a character is painted.


Now all that we need to do in order to have this feature selectable at run-time, is to implement a new sysctl adjustment, and ensure that rasops_init_devcmap is called when it's value is changed.


Implementing a new sysctl


Readers who are not familiar with the kernel internals might naively assume that there is a simple library function that can be called with the name of the sysctl value we want to read, and which will return the result. Alas, things are not quite that straightforward, but nevertheless, implementing a new sysctl is not particularly difficult once you know the procedure.


The manual page for sysctl_int will give you an idea of the complexity of the sysctl interface, but don't worry if you don't understand it.


Our new sysctl will be `kern.exotic', and it will contain an integer value. This will allow us to select the default colour pallette by setting it's value to 0, or any number of alternative pallettes with other settings.


The first file we need to edit is sys/sys/sysctl.h, which contains the list of identifiers in the kern hierarchy. The last entry in this list should be KERN_MAXID, which indicates the largest number in use. As of OpenBSD 7.1-release, this is set to 90, (the most recent addition was kern.video.record, made during the development cycle of OpenBSD 6.9). We just need to increase the value of KERN_MAXID to 91, and add our own definition as 90:


 #define KERN_EXOTIC        90
 #define KERN_MAXID         91

Immediately after this, we should add our new sysctl to the end of CTL_KERN_NAMES:


 { "exotic", CTLTYPE_INT }, \

At this point, if we were to re-build the kernel, then the definitions of the new sysctl would be compiled into it. However, we will also need to re-compile the userland utility /sbin/sysctl in order for it to recognise the new sysctl that we've added, and that utility includes the sysctl.h header file from /usr/include/sys/sysctl.h, rather than from it's location within the kernel source, so we need to make sure that our changes are reflected there as well:


 # cp -p /usr/src/sys/sys/sysctl.h /usr/include/sys/

Then we can re-compile and re-install /sbin/sysctl:


 # cd /usr/src/sys/sbin/sysctl
 # make
 # make install

With these changes in place, and rebooted into a freshly compiled kernel, we can now access our new sysctl, and change the value stored in it:


 # sysctl kern.exotic
 kern.exotic=0
 # sysctl kern.exotic=1
 kern.exotic: 0 -> 1
 # sysctl kern.exotic
 kern.exotic=1

Making use of our new sysctl


Of course, our new sysctl doesn't actually do anything yet. For that we need to add code to sys/kern/kern_sysctl.c.


The easiest way to make use of an integer sysctl value in the kernel is to create a globally-scoped integer variable which will mirror it's value. In other words, when a new value is set from userland using /sbin/sysctl, that value will update the global variable.


We'll call our global variable `exotic', and define it in sys/dev/rasops.c, at the beginning of the file just after the includes, with a simple:


 int exotic=0;

Now, we need to add a couple of lines to the large switch statement, in function kern_sysctl, in file sys/kern/kern_sysctl.c:


 case KERN_EXOTIC:
         return (sysctl_int(oldp, oldlenp, newp, newlen, exotic));

Where KERN_EXOTIC is the name we defined earlier in sys/sys/sysctl.h, and &exotic is a reference to our new global variable.


This is about the simplest way to implement a new syctl, and will allow us to set kern.exotic to any integer value, which can then be used to control something elsewhere in our code. If we wanted to restrict the range of values that kern.exotic could be set to, we could add more code to our new entry in the switch statement, but usually it's just easier to interpret unused values as if they were 0 in our own functions.


Our code to change the actual colour values can now be made conditional on the value of the global variable:


 if (exotic==1) {
         asm volatile ("shrb %%al;" : "+a" (c) : :);
         }

At this point, we're almost done. Now we just need to make sure that rasops_init_devcmap is called after updating kern.exotic.


Calling rasops_init_devcmap when our sysctl is updated


Currently, rasops_init_devcmap is only being called once, at device initialisation time, right at the end of rasops_init. This was fine for our quick test, where we just inserted one line to adjust the colour values permanently, but if we want to control the effect via our new sysctl then we'll need to call rasops_init_devcmap whenever it's updated.


Unfortunately, we can't easily call rasops_init_devcmap directly from our new sysctl handler in sys/kern/kern_sysctl.c, as we need to pass it a pointer to the correct rasops_info structure. One way around this would be to add a second global variable to act as a flag to show that the value had been changed, and then to check this flag every time rasops32_putchar, (or the putchar function for any other bit-depth), was called.


This works, but it seems somewhat wasteful to be checking the flag every time a character is written to the framebuffer.


A reasonable trade-off in terms of code complexity and performance, is to add a call to rasops_init_devcmap to rasops_eraserows. This function is called fairly frequently, for example during scrolling, when clearing the screen, or switching to a different virtual terminal, but still usually much less often than the various putchar functions.


Re-initialising the device colour map from here does cause a slightly unusual visual effect, though. If the cursor is not already at the bottom of the screen, and so the screen doesn't scroll when the sysctl command is entered, the next line of text will not immediately be displayed with the new pallette, since we haven't yet called rasops_init_devcmap to update it. A simple switch to another virtual terminal and back, or clearing the display will cause the screen to be re-painted using the new pallette, though, and the trade off seems worthwhile for the lower performance overhead.


Of course, wherever we call rasops_init_devcmap from, existing text on the screen will not automatically be re-painted. It will remain on the display in the old colour pallette until it's re-painted manually, usually either by a scroll, or by switching VTs. If we wanted the whole display to be automatically updated, we would need to re-write the contents of each character cell after loading the new colour pallette.


So, to call rasops_init_devcmap from rasops_eraserows, we just need to add the following line immediately after the assignment to variable ri:


 rasops_init_devcmap (ri);

Then re-compile the kernel, and re-boot, ready to test the new feature!


Testing the new feature


By default, when we boot into the new kernel we are initially greeted with the standard colour scheme.


All we need to do to see the alternative colour scheme is to log in as root, set kern.exotic to 1, and clear the screen:


 # sysctl kern.exotic=1
 kern.exotic: 0 -> 1
 # clear

And there we are! Late night hacking sessions just got more comfortable.


Adding additional alternative pallettes


Now that we have the basic functionality working...


We can easily add some extra alternative colour pallettes by replacing the if statement with a switch, and simply applying different mathematical transformations to the red, green, and blue values. This is nicely illustrated by the following block of code:


 #define RED ((c & 0xff0000) >> 16)
 #define GREEN ((c & 0x00ff00) >> 8)
 #define BLUE (c & 0x0000ff)
 #define GREY ((int)(((GREEN*0.7)+(RED*0.2)+(BLUE*0.1))))
 switch (exotic) {
         case 1:
                 /* Reduce blue component */
                 #if defined (__amd64__) || defined (__i386__)
                 asm volatile ("shrb %%al;" : "+a" (c) : :);
                 #else
                 c=(c & 0xffff00) | ((c & 0xff)>>1);
                 #endif
                 break ;
         case 2:
                 /* Convert to green-scale */
                 c=(int)(((GREEN*0.7)+(RED*0.2)+(BLUE*0.1)))<<8;
                 break ;
         case 3:
                 /* Convert to amber-scale */
                 c=(GREY<<16)|(GREY<<8);
                 break ;
         case 4:
                 /* Convert to greyscale */
                 c=((GREY<<16) | (GREY<<8) | GREY);
                 break ;
         case 5:
                 /* Convert to pink-scale */
                 c=((GREY<<16) | ((GREY>>1)<<8) | (GREY>>1));
                 break ;
         case 6:
                 /* Convert to greyscale and reduce blue component */
                 c=((GREY<<16) | (GREY<<8) | (GREY>>1));
                 break ;
         default:
                 break;
 }

This provides us with a choice of no less than seven different operating modes, when inserted into rasops.c just before the final assignment of c to the device colour map array entry in function rasops_init_devcmap:


0 Normal

1 Night mode, (blue light reduced)

2 Greenscreen monitor simulation mode

3 Amber phosphor monitor simulation mode

4 Greyscale mode

5 Shades of pink

6 Night greyscale mode, (yellow tinted greyscale)


In this example code, we also see how to include architecture-specific code into the kernel by using the C pre-processor to test for architecture specific defines. Here, we are including the X86 assembly code on the amd64 and i386 platforms, but using equivalent C code on all others.


A side project - adding support for the dim attribute


The framebuffer console on OpenBSD supports a large number of control characters and escape sequences, such as the cursor positioning functions used by vt-100 terminals, as well as ANSI escape sequences for colour. However, one particularly useful escape sequence is missing from the emulation, and that is the dim attribute. Since we're looking at colour reproduction on the framebuffer console, we might as well take a few minutes to add support for the dim attribute.


This escape sequence works similarly to the bold attribute, which the OpenBSD framebuffer console renders as high-intensity. For dim text, we'll render the characters at a lower intensity.


 [ESC][2m        Set dim mode

The following shell script will output text with four sets of attributes: dim, bold and dim together, normal, and bold:


 #!/bin/sh
 echo "[ESC][0m[ESC][2mDIM"
 echo "[ESC][0m[ESC][1m[ESC][2mBOLD AND DIM"
 echo "[ESC][0mNORMAL"
 echo "[ESC][0m[ESC][1mBOLD[ESC][0m"

Run on an unmodified OpenBSD system, the output will only show two different intensity levels visible. After our changes we will see four intensity levels:


 DIM
 BOLD AND DIM
 NORMAL
 BOLD

The actual dimming code


To display a colour at half brightness, we can simply shift each of the red, green, and blue channels by one bit, dividing their values by two.


Assuming that we are dealing with 24-bit RGB data, the code would look something like this:


 red=((f & 0xff0000) >> 16);
 green=((f & 0x00ff00) >> 8);
 blue=((f & 0x0000ff));
 red=red>>1;
 green=green>>1;
 blue=blue>>1;
 f=((red<<16)|(green<<8)|blue);

This can be written more efficiently as:


f=(f>>1) & 0x007F7F7F;

In this case we are simply shifting a whole 32-bit value one bit to the right, and masking the high bits of each of the lower three bytes to avoid shifting a value from the neighbouring byte into them.


Since the dim attribute can be applied on a character by character basis, this time it actually makes logical sense to put the supporting code in the rasops putchar routines. The above line can be inserted directly in to rasops32.c immediately after the value of variable f is assigned from the device colour map array.


Note that we are only dimming the foreground colour in the example above, and the background stays at it's usual brightness. Of course, it would be trivial to dim the background colour as well, we would just need to do the equivalent bit shift for variable `b'.


If we now compile a new kernel with this additional code and reboot into it, we will see all of the text displayed at half brightness...


Adding a new escape sequence to the terminal emulation code


Most of the code to handle parsing of terminal escape sequences is in sys/dev/wscons/wsemul_vt100_subr.c, and this is where we will add our code to recognise [ESC][2m.


This escape sequence is one of a family which all begin with the CSI, (control sequence introducer), or [ESC][. The function that handles these is wsemul_vt100_handle_csi.



Fun fact!


Although the two byte sequence of [ESC][ is by far the most common way to signal CSI to an ANSI compatible terminal, it was actually always intended as a legacy 7-bit compatible equivalent encoding, and the single byte 8-bit code 0x9b was also defined as CSI. However the 8-bit code never gained much popularity, and as 8-bit extensions of ASCII began to emerge encoding printable characters with the same code it's use became increasingly impractical.



Within the CSI family of escape sequences are a set of SGR, (select graphic rendition), sequences, which all have a common format of the CSI sequence, followed by one or more parameters and ending with `m':


 [ESC][ PARAMETERS m

Looking through the code for wsemul_vt100_handle_csi, we can see that it mostly consists of a single large switch statement checking the arguments of the CSI sequence, and that case `m' is indeed commented as SGR. This then leads to a nested switch statement to parse the SGR parameters.


Most of the SGR sequences, with the exception of the reset and colour selection sequences, simply set or reset bits in the flags variable. These flags are later parsed by the rasops code in rasops.c, in the function rasops_pack_cattr.


The bits of the flags variable are defined in sys/dev/wscons/wsdisplayvar.h, and all begin with WSATTR_. As of OpenBSD 7.1, only bits 0 - 4 have definitions, leaving us free to use the other bits for our own purposes.


For the dim flag, we'll use bit 5:


 #define WSATTR_DIM        32

With this in place, we can add a new case to the SGR switch statement in wsemul_vt100_handle_csi:


 case 2: /* dim */
 flags |= WSATTR_DIM;
 break;

We should also modify case 22, which currently clears the bold attribute, so that it clears our new dim attribute as well, either by adding an additional bitmask operation:


 flags &= ~WSATTR_DIM;

Or by replacing the existing code with a combined bitmask for the two flag bits:


 flags &= ~(WSATTR_DIM | WSATTR_HILIT);

Passing the new flag bit through the rasops code


The WSATTR_ flags are parsed by the function rasops_pack_cattr in rasops.c, (or more correctly, whichever function is pointed to by the function pointer ri_ops.pack_attr, which could be rasops_pack_mattr in the case of a monochrome display).


This function prepares the attr value which is eventually passed to the relevant putchar function in the bit-depth specific rasops code. The attr value stores the index to the background colour in bits 16-19, the index to the foreground colour in bits 24-27, and three flags in bits 0-2. These flags are completely different from the WSATTR_ flag bit definitions, and don't actually seem to be defined as mnemonics but are instead referenced in the code as magic numbers without much explanation as to what they do.


Bit 0 indicates underlining, whilst bits 1 and 2 indicate that the foreground and background colours respectively are a shade of grey, in other words, that the red, green, and blue components are equal.


We'll use bit 3 to indicate that our dim attribute is active. The existing code re-purposes the variable `flg' from representing the bitmap of flags supplied to the function, to storing the new flags to put in the lower bits of `attr'. Since it only has to deal with a single incoming flag being passed onwards, the code simply uses the tertiary operator to set or reset bit 0, based on the value it already contains.


Unfortunately, this then destroys the contents of the bit that we are using to store our dim flag, so we either need to re-write this existing code, or simply store our dim attribute bit elsewhere and insert it into `attr' afterwards.


By this point in the function, the `swap' variable is no longer used, so we can re-purpose it to hold our flag temporarily:


 swap=((flg & WSATTR_DIM)==WSATTR_DIM ? 1 : 0);

Then add it back after the underline flag code:


 flg |= swap;

With these changes, our dim attribute will be passed to the purchar routine as part of the attr parameter.


Making the rasops dimming code conditional on the dim flag


All we need to do now is to make our dimming code in rasops32_putchar conditional on bit 3 of attr being set:


 /* Implement dim if bit 3 of the attribute is set */
 if ((attr & 8)==8) {
         f=(f>>1) & 0x007F7F7F;
         }

Re-compile, then re-boot into the new kernel, and we're done! Now our small test program above correctly displays the four combinations of normal, dim, bold, and bold with dim.


Implementing strikethrough, double underline, and other features


As we have just shown, implementing the parsing of a new escape sequence and it's corresponding graphical effect is relatively straightforward. This is especially true if we are only concerned with supporting 32 bpp displays.


The graphical effects for strikethrough and double underline are trivial to implement as they are basically just variations on the regular underlining code.


Italic text could easily be simulated by right-shifting each row of pixel data a suitable number of bits to the right.


Blinking text would be more challenging, as this would require on-going updating of the bitmapped display for characters that had already been painted.


Ready made patches


For those readers who are not interested in learning how this all works and would rather just see the end result, we've produced patch sets implementing the features. Patch sets are currently available against OpenBSD 7.0 and OpenBSD 7.1.


Patch format: Unified diff with signature
Patch size:   6590 bytes
Patch hash:   SHA512 (candlelit_console_patch_7.0.sig) = aXsbCKdjuUOlcK/d4SNIKt4ZI+wCY8LZ8KjGR27P068cBUFPorW5rv0CnMTiyWSQWssV+ckkPJh+FtGpB4i2TQ==
Patch hash:   SHA512 (candlelit_console_patch_7.1.sig) = +4GvR/Sw1pSzWoPDt2zAJK4qRbfjfdPMsH9Hdn2hv2wpNQO3wrp+h+IhZoJLmyQXCcdVpXIyVPPCyEUfoJVqGw==

Two diffs are available, one applies cleanly to OpenBSD 7.0-release, and the other is available against OpenBSD 7.1-release.


Downloads

candlelit_console_patch_7.0.sig - Kernel patch with embedded signify signature

candlelit_console_patch_7.1.sig - Kernel patch with embedded signify signature

colour_chart - Shell script to display a colour test chart


> THE ABOVE LINKED SOFTWARE IS PROVIDED 'AS IS' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL EXOTIC SILICON BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THE SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


In addition to the features explained in this article, the version of the code in the patchsets also implements double underlining and strikethrough!


Code compatibility with NetBSD


Although the examples and commentary in this article have been based on the OpenBSD kernel, the broad concepts can quite easily be applied to NetBSD as well. The two codebases are fairly similar in most of the areas that we have touched, the main caveats being:


The changes we've made to rasops_init_devcmap in rasops.c will apply almost unchanged to NetBSD as well, since the functions are very similar and even use the same variable names. If we are only interested in supporting 32 bpp displays, the changes are actually trivial to make in NetBSD, as we can just put our code to modify the value of 'c' at the end of the case 32 condition of the switch statement.

Adding sysctls is somewhat different.

The individual rasops??.c files are quite different, as the putchar routine is genericified and included in rasops_putchar.h, rather than being written separately for each bit-depth.

Underlining uses two values, ri_ul.height, and ri_ul.off, to set the height and offset of the underline based on the font size. In OpenBSD, these values are hard-coded.

The equivalent function in NetBSD to the rasops_pack_cattr function in OpenBSD is rasops_allocattr_color. This function is very similar to rasops_pack_cattr in OpenBSD, but doesn't set bit 1 of attr for underline. Instead, it passes through bits 0 - 11 from the supplied value for flg, including bit 3 which is defined by WSATTR_UNDERLINE.


Summary and suggested programming exercises


In this article, we've seen how to implement a new sysctl knob in the OpenBSD kernel and use it to select between a choice of several different colour pallettes for the framebuffer kernel. We've also looked at the kernel code that parses terminal escape sequences, and added a new one.


Plenty of scope exists for making further enhancements to the code. Some ideas for additional features include:


Implementing additional alternative colour pallettes.

Implementing more attributes such as blinking, and italics.

Expanding support for bit depths other than 32 bpp.

Implementing direct selection of 24-bit RGB colours via escape sequences.


Home page of the Exotic Silicon gemini capsule.

Your use of this gemini capsule is subject to the terms and conditions of use.

Copyright 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023 Exotic Silicon. All rights reserved.

-- Response ended

-- Page fetched on Sat May 11 08:26:03 2024