Web: Packaging assets to a binary the web way

We have a game in html with a bunch of images and sound effects. Instead of having a bunch of requests, we want to package all the asset files into a single binary, download it and extract the files. Something like tar/zip but... been there done that.

So lets follow the ancient philosophy of a header and payload. To be more exact: magic bytes, version, header length, (for each file) path size, path, file size and all the file contents concatenated in the end.

Lets use 4 byte integers for all to simplify life, and we have the ArrayBuffers and DataView ready to implement a C struct like parser.

Ok lets start with the magic bytes.... but wait a second this is the web, is packing bytes really the thing that we need? I mean just downloading the file will take couple of orders of magnitude of time more than parsing it. So lets do the exact opposite of packing bytes: lets do JSON!

Lets put all the header data in a JSON object and then we have: magic bytes, header size, JSON dump, file contents. Yeah... but it could be nice to just have the JSON object and then the file contents.

What is the most used operation in the world of js: split once. All we need is a delimiter, what is never used in a JSON object: all the non printable characters, lets take the first the null byte (0x00).

So we have it: JSON header object, byte 0x00, file contents.

Which if you think about its agnostic to the actual content, you can put whatever you like in the header and whatever you like in the binary part.

I shall name it JHBF (json header binary file).

done_

Techniques: Dev inplace replacement code

You have your utils.js with the classic document.querySelector shortcut:

function q(selector) {
    return document.querySelector(selector);
}

Lets say that the "assertion" is that this should be only called when we know the selector will always match a single element (for the other cases we simply call other functions)

We leave this "efficient" version, and we replicate the an error checking version of it in a new file utils.dev.js:

function q(selector) {
    const res = document.querySelectorAll(selector);
    if (res.length == 1) return res[0];
    throw `Found ${res.length} matches for '${selector}'`;
}

And then in the "build system" we specify one of the files based on the release mode.

done_

Techniques: From code to data and back to code

Lets create a series of fields:

{
  container.add(new Field("name", "text"));
  container.add(new Field("surname", "text"));
  container.add(new Field("something", "text"));
}

But wait we can just collect all the actuall info and then have the data so the code 'is not repeating':

fields = [
  { title: "name", type: "text" },
  { title: "surname", type: "text" },
  { title: "something", type: "text" },
];

But wait we can use code so the data 'is not repeating':

fields = [
  ...[ "name", "surname", "something" ]
       .map(i => ({ title: i , type: "text" }))
];

But wait we can use data so the code 'is not repeating':

fields = [
  { titles: [ "name", "surname", "something" ], type: "text" },
  { titles: [ "birth", "death" ], type: "date" },
].flatMap(magic);

But if you really thing about it, and ignore any "non-programmers", who are those guys anyway, the first one is what actually what happens, and by definition has no limitations because it's actually the thing itself, like C. And then you add some sugar and you are also modern:

{
  container.add(
    field.text("name"),
    field.text("surname"),
    field.text("something"),
    field.date("birth"),
    field.date("death").optional,
  )
}

done_

Techniques: Multiple localhost sessions

Lets assume you have an web app with some sort of login system, and you are testing things on a local web server bound to "localhost" and you want to test multiple logged in users at the same time.

We start by creating multiple "domains" pointing to the localhost via the /ets/hosts file:

127.0.0.1  localhost1
127.0.0.1  localhost2
127.0.0.1  localhost3
127.0.0.1  localhost4
127.0.0.1  localhost5
127.0.0.1  localhost6
127.0.0.1  localhost7
127.0.0.1  localhost8
127.0.0.1  localhost9

Because the session and local storage is instanced per domain name (or even the cookies for the old school) using the localhostX will each be isolated.

Then you fight with your web server that definitely will not like requesting http with something different than localhost, but you can win.

And then you fight with the OAuth callback redirect url and... you lose the fight... and you go back to the firefox containers.

done_

Techniques: Patch files

Create a patch file:

• Locate the file we want to change, eg. path/file.json
• Create a copy of the file and edit it in the same directory, eg. path/file.new.json
• Create the diff file by running the program diff, eg.

  diff -u path/file.json path/file.new.json > file.json.diff

• Ensure or change the first lines of the diff file in order to point to the desired files, eg. path/file.json to build/path/file.json
• Apply the patch by running the program patch, eg.

  patch -p0 < file.json.diff

Notes:

• Both programs, diff and patch, should be part of the core programs on any linux distribution
• The -u option specifies that the created file is of unified type
• The -p0 options specifies that the full paths inside the first lines of the diff file should be used

done_

JavaScript: Generic selector proxied object

Lets create a thing to replace the very simple selector calls for id and classes.

First lets have simple selector for both the id and single class.

return document.getElementById(name) || 
    [...document.getElementsByClassName(name)]

Here we also return a single value when you pass an id and an array (expanding the default HTMLCollection) when a class is passed.

Now, if we want the make the whole thing more elegant like this:

elements.footer // == document.querySelector('#footer')
elements.button // == document.querySelectorAll('.button') 

We can use the a proxy handler to catch the selector like text that we defined above and return the elements.

const elements = new Proxy({}, {
    get(target, name) {
        return document.getElementById(name) || 
        	[...document.getElementsByClassName(name)];
    }
})

Where every time a property of elements must be deference the name of the property will end up in the get method in the name argument. 

_done

LibGDX: Blending in html gwt web

If you use OpenGl blending functions, you probably are doing something with the alpha buffer (or not). By default the alpha buffer is disabled on the html / gwt / web config.

So you need to go to the HtmlLauncher and set the alpha boolean to true:

public GwtApplicationConfiguration getConfig() {
	final GwtApplicationConfiguration config = new ...;
	config.alpha = true;
	return config;
}

A side effect by enabling the alpha buffer is the actual background will become pseudo transparent. At the end of the drawing the alpha buffer will be handled as the alpha channel of the whole canvas. So if there is no need for some transparency, at the end of drawing the whole alpha buffer should be filled with 1s. 

done_

Html: Unwanted cached resources

So as you write a simple a html file with one or two resources, lets say one css and one js file, and you then deploy it (= copy paste) to a sever, and then you make some changes (= fix a bug), and then you redeploy (= copy paste, overwrite popup, press yes), and then nothing changes.

The browser, like always, remembers the old 'main.js' from before and then you pres Ctrl+F5. But your tester in the other side, has no clue what a Ctrl is. So you have to do something.

The first thing is to add a simple '?1' at the end of 'main.js' and every other resource, the browser thinks it may be a different file and deletes the cached version.

So now lets make a script to automate the replacement of the '?1' with '?2':

sed 's/"\(.*\)?.*"/"\1?2"||/' -i index.html

But we may want the browser to actually keep some cached files, the files that have not changed. Only if, instead of '?2', we could generate a number that only changes if the actual file changes and also does not repeat it self.

Hash functions? Checksums? Yes.

Lets use 'md5sum' to freak out the people who don't know. A single sed will not make it, lets write a sed to generate multiple seds and pipe it into a sh.

First we find checksums:

find -regex '.*\(js\|css\)$' -exec md5sum {} \+

Then we pipe that to our first sed:

sed 's/\(.*\)  \(.*\)/ ... \2 ... \1 ... /'

where in the place '...' we reconstruct version of our original sed (gets very unreadable). And finally we pipe it to a sh to actually do the substitution.

The whole thing if placed in Makefile would look like this:

index.html: *.css *.js
    find *.css *.js -exec md5sum {} \+ | \
        sed 's/\(.*\)  \(.*\)/sed \"s|\\"\2?[^\\"]*|\\"\2?\1|\" -i index.html/' | sh

_done

Javascript: Self call re-call interval set

 So we have the requestAnimationFrame, you pass a function and it called after a very specific while (irrelevant to the following).

Now lets see the basic structure:

function loop() {
    update()
    requestAnimationFrame(loop)
}
loop()

Ok now lets wrap the function and call it without calling it by name:

(function loop() {
    update()
    requestAnimationFrame(loop)
})()

Is there any way to remove the name of function and still pass it? There is a magical object called arguments which appears out of nothing every time a function is executed. And the member that we need is arguments.callee which is self reference to the function being executed. So we get rid of the name:

(function () {
    update()
    requestAnimationFrame(arguments.callee)
})()

And of course to maximize the coolness we put in a single line:

(function() { update(); requestAnimationFrame(arguments.callee) })()

Ok, this very cool, but is it worth it? Let's analyze what this code succeeds to do:

1. Confuses the next reader of the code.
2. Confuses also the 2nd reader.
3. Confuses also the 3rd reader.
...
N. Confuses also the N-th reader.

So you have to choose what is cooler: confusing no people or confusing up to infinite people? Comment below, like and subscribe.

done_

Techniques: Build-time pre-package capture

 Usually when things/files are packaged from simple files to single monstrosity, you no longer have the ability to ls/dir to lookup what things/files exist.

An example is a Java jar, where to list for example all the packaged files you either cannot or you need to open a zip stream and figure out yourself.

Also in Java and other languages with limited introspection you cannot for example list all the classes that implement the interface X.

So:

Create a script than generates all the information you want.

Put that script in the build/compile process.

Include the file in the project.

Write some code to parse the file.

Example of listing all the inheritance properties by finding all the extends keyword:

grep -hr extends | sed -E 's/.*class (\w+) extends (\w+) .*/\2 -> \1/'

Parent -> Child1
Parent -> Child2
...

No need to say again and again "This is not supported, I cannot do it", supported yourself, you are a damn programmer! 

done_ 

Java: Return enum instead of boolean

We have a game where you place down buildings. So there is always a method that looks like that:

public boolean validPlace(BuildingType type, Location location) {

    for (int ix = 0; ix < type.w; ix++) {
        for (int iy = 0; iy < type.h; iy++) {
            
            if (/* check if overlapping */) {
                return false;
            } else if (/* check if road missing */) {
                return false;
            }
        }
    }
    return true;
}

The problem: While writing the game logic everything is fine, but then comes the UI. A can palce / cannot place is not enough for a good UI, you need the reason. And you usually have another method that finds out the reason. But can we use the same method?

First we need an enum with all the possible problems:

public enum PlacementValidity {
    Ok,
    Overlapping,
    RoadMissing;

    public boolean ok() {
        return this == Ok;
    }
}

Then we just return the enum instead of the boolean:

public PlacementValidity validPlace(BuildingType type, Location location) {

    for (int ix = 0; ix < type.w; ix++) {
        for (int iy = 0; iy < type.h; iy++) {
                
            if (/* check if overlapping */) {
                return PlacementValidity.Overlapping;
            } else if (/* check if road missing */) {
                return PlacementValidity.RoadMissing;
            }
        }
    }
    return PlacementValidity.Ok;
}

The helper function ok() method can also be used to reduce verbosity on the game logic side of the code.

if (validPlace(/* ... */).ok()) {

done_

C: Autogen headers

It is known that I don't like writing header files and forward declarations. The keyword is writing, I like what you can do with header files, however I would also like to have an option, some times, to just autogen the headers. Also it could be nice to have a header file even in a single file program just to avoid forward declarations.

Lets thing about it... ok you need a C parser because C can be crazy some times. You can have macros and also you can have all kinds of whitespace in a function declaration. So you either write a program witch handles everything or you write a one-line bash thingy to handle only your style of code.

Lets do the second one (obviously). I write functions like this:

int send(cost char* text) {

So we grep all the lines which start with a character and end with a open bracket and then we replace the bracket with a semicolon:

grep '^\w.* {$' | sed 's/ {/;/'

Ok, lets put it in the Makefile:

%.h: %.c
    cat $< | grep '^\w.* {$$' | sed 's/ {/;/' > $@

What about the structs? Done, already handled.
What about typedefs? Nope... next post?

And a bonus effect is that if you include the header of the source file in its self you no longer need forward declarations.



done_

Java: Let's serialize with a map

Lets just use a map. So we have a timer like thing:

public class Time implements Serializable {

    private long duration;
    private long start;

    // crazy things

}

Now lets put everything in a map:

    private void writeObject(java.io.ObjectOutputStream out) throws ... {
        Map map = new HashMap<>();

        map.put("duration", duration);
        map.put("start", start);
        out.writeObject(map);
    }

Now lets get everything back from the map:

    private void readObject(java.io.ObjectInputStream in) throws ... {
        Map map = (java.util.Map) in.readObject();

        duration = (long) map.get("duration");
        start = (long) map.get("start");
    }

Lets also put this because we are handling the versioning:

    public static final long serialVersionUID = 1L;

However this is too much stuff to write, so lets make a script to write it for us: https://maanoo.com/testing/JavagenSerialMap.html

Now lets add a new thing to see how to handle versioning

    private TimeUnit unit;  // v.2

So we have to update the things: (our script that is)

    private void writeObject(java.io.ObjectOutputStream out) throws ... {
        Map map = new HashMap<>();

        map.put("duration", duration);
        map.put("start", start);
        map.put("unit", unit);
        out.writeObject(map);
    }

    private void readObject(java.io.ObjectInputStream in) throws ... {
        Map map = (Map) in.readObject();

        duration = (long) map.get("duration");
        start = (long) map.get("start");
        if (map.containsKey("unit")) {
            unit = (TimeUnit) map.get("unit");
        } else {
            // TODO unit
        }
    }

And we replace the TODO by setting the default value of the unit:

    unit = TimeUnit.Milliseconds

Also, if or when we are sure that there is none of the old version left, we can remove the check of the key and remove the setting of the default value.

Except the full control that this system provides, the actual data that are stored are serialized primitives and collections, making possible to be access and/or debug the saved data by external program.



done_

Java: Optional or just null

So I was making a simple game on the weekend, as we do, and there was a simple (again) player class:

public class Player {
    public final String name;
    private Team team;
    ....
    public boolean friendly(Player other) {
        return team.has(other);
    }
}

But the obvious thing is, can it be null? Yes it can, in cases where the player does not have a team. So I said 'Let's try Optional, why not`.

    private Optional<Team> team;

Then the possibilities for the return statement where limitless from the most straight forward to the most elegant:

    return team.isPresent() && team.get().has(other)

    return team.map(i -> i.has(other)).orElse(false);

    return team.filter(i -> i.has(other)).isPresent();

    return team.orElse(Team.EMPTY).has(other);

    

Or I could remove the Optional again and have just the old thing:

    return team != null && team.has(other);

But 'How will you remember that it can be null in other places?' said the guy. Its a private field, how many places are there really. Also it is logical to null so I would probably check it anyhow.

done_

Javascript: Do it like php!

So, I just wanted a simple thing... I had a static html page where I wanted to put some things in dynamically: a simple date at the start and an extended date in the middle of the text. (Also what is server side?)

Pure elegant javascript:

.... <span data-value="simple"></span> ....

<script>

var data = {
    simple: gen_simple(),
    ....
}

for (var i of document.querySelectorAll('[data-value]')) {
    i.innerHTML = data[i.getAttribute('data-value')];
}

<script>

Php infected javascript:

.... <script>document.write(gen_simple())</script> ....

plus if in 'I was never here' mode:

....
<script>
for (var i of document.querySelectorAll('script')) i.parentNode.removeChild(i);
</script>

done_

Java: Iterable iterator

We all have been in a position with an Iterator in our hands and trying to pass it to a for each loop. And then for some reason that is totally crazy for a second an error pops up:

"Can only iterate over an array or an instance of java.lang.Iterable"

What... a ok... and then what? You could just fix your logic in order or just google for "java iterator to iterable". But could you just make the iterator also an iterable somehow.

Step one, obviously, add the Iterable interface to your iterator:

public class SuperIterator<T> implements Iterator<T>, Iterable<T> {

Step two, add the missing method by return itself as its iterator:

@Override
public Iterator<Character> iterator() {
    return this;
}

done_

Java: enum or boolean for two value properties

The is always that tip that floats around that you should always use enums instead of booleans, even when there is only two values.

Of course the point is that it will be much easier to read and mainly to extend in the future.

So from a simple

 public void setVisible(boolean value) { ...

you go to

 public enum Visibility { Visible, Gone }

 public void setVisible(Visibility value) { ...

And in the future you could add eg. the Invisible value.

But you miss the simplicity of a boolean and you also get the verbosity of an enum

 setVisible(Visibility.Visible)

and this happens

 setVisible(value ? Visibility.Visible : Visibility.Gone)

The classic solution is to add methods like show() or hide() to avoid the use of the enum

However a simply solution to collect the benefits of both world:

 public void setVisible(boolean value) {
     setVisible(value ? Visibility.Visible : Visibility.Gone)
 }

 public void setVisible(Visibility value) { ...

where the two boolean values will correspond to the two most used values of the enum

And why not add and some methods for the most used values

 public void show() {
     setVisible(Visibility.Visible)
 }

So in the end you have everything

 show()
 setVisible(true)
 setVisible(Visibility.Visible)

done_

C++: Euclidean distance variations

So how about this new std::hypot?

 float dx = x1 - x2;
 float dy = y1 - y2;
 return std::hypot(dx, dy);

Time: 12139783 ticks

How about the old one std::sqrt?

 float dx = x1 - x2;
 float dy = y1 - y2;
 return std::sqrt(dx*dx + dy*dy);

Time: 7026125 ticks (1.7 times faster)

Oh, the std::hypot performs an overflow check, so if you don't care about that just use std::sqrt

And of course the very old std:: nothing

 float dx = x1 - x2;
 float dy = y1 - y2;
 return dx*dx + dy*dy;

Time: 5667812 ticks (2.1 and 1.2 times faster)

Wait, can I just do

 return (x1 - x2)*(x1 - x2) + (y1 - y2)*(y1 - y2);

and let the optimizer fix it. Lets check

Time: 5592648 ticks (seems the same)


Source


done_

C++: New keyword exceptions

When new is used, a bad_alloc can be thrown. To return null pointer on error instead of the exception a second format can be used:

Site *maanoo = new (std::nothrow) Site("maanoo.com");

done_

Techniques: Random art with symmetry

So lets add many circles together:

Thats ok but what if we add some symmetry.

Lets add symmetry from one axis:

From two axis:

Those examples are random but the point is that have something not random, they have structure.

I like the more complex symmetry from the center point (not the math definition):

continue_