Optimizing resource downloads using a caching proxy

Server builds starting at pipeline ID 1679 added the ability to configure the fileServer field used for resource downloading using a set of commands:

# Set the file server for the specified resource regex.
# The URL should *not* end with a slash.
fileserver_add ".*" "http://10.10.0.1/files"

# Remove the file server associated with a resource regex.
fileserver_remove ".*"

# List all registered file server patterns.
fileserver_list

# Old command, but **required** to not get corrupted cache entries.
adhesive_cdnKey "someSecurePassphrase"

There’s currently no cache invalidation logic based on hashes (this’ll need yet another server update due to the file server not ignoring query strings + a client update), so make sure to clear your proxy’s cache before you modify/restart a resource.

Here’s an example NGINX configuration for setting this up (but you should, for example, run this behind yet another proxy which offers HTTP/2 over TLS, or otherwise set this up properly – we’re expecting the community to provide fleshed-out examples on the forums):

default.template

proxy_cache_path /srv/cache levels=1:2 keys_zone=assets:48m max_size=10g ;
log_format asset '$remote_addr - [$time_local] "$request" $status $body_bytes_sent $upstream_cache_status';

server {
        listen 80;

        location /files/ {
                access_log /dev/stdout asset;
                add_header X-Cache-Status $upstream_cache_status;
                proxy_cache_lock on;
                proxy_pass $REMOTE$request_uri;
                proxy_cache assets;
                proxy_cache_valid 1y;
                proxy_cache_key $request_uri$is_args$args;
        }
}

Dockerfile

FROM nginx:alpine
COPY default.template /etc/nginx/conf.d/default.template
CMD sh -c "envsubst \"`env | awk -F = '{printf \" \\\\$%s\", $1}'`\" < /etc/nginx/conf.d/default.template > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"

Start command

docker build -t fxproxy .
docker run -d --name=fxproxy -e REMOTE=http://10.10.0.2:30120 -p 80:80 -v $PWD/cache:/srv/cache fxproxy

OneSync: intercepting game events (such as explosions)

The latest version of the server (1543+) added support for parsing and preventing routing of game events. This currently only supports parsing CExplosionEvent, but this might be a fairly useful one to prevent routing of ‘excessive’ explosions, or explosions that are ‘too close’ to a player and not of the correct weapon type.

Here’s an example:

-- SERVER script, requires OneSync!
AddEventHandler('explosionEvent', function(sender, ev)
    print(GetPlayerName(sender), json.encode(ev))
end)

This’ll show JSON data similar to the following:

{"explosionType":0,"isAudible":true,"posX":742.84313964844,"cameraShake":1.0,"isInvisible":true,"ownerNetId":0,"posY":-1808.2889404297,"damageScale":1.0,"posZ":33.105224609375}

If you want to, say, make an explosion-free zone:

AddEventHandler('explosionEvent', function(sender, ev)
    if ev.posX > 2000.0 and ev.posY > 2000.0 and ev.posX < 3000.0 and ev.posY < 3000.0 then
        CancelEvent()
    end
end)

This can be used in a variety of ways – rate limiting, automatic warning/kicking, disabling routing of bad explosions, etc.

Two new experimental OneSync variables

In the latest Windows server release (1504 or higher), two experimental variables have been added to help troubleshooting problematic behavior and offer a ‘workaround’ for common problems.

onesync_distanceCullVehicles true

This console variable (set it in your config, or in the console at runtime) will lead to player-occupied vehicles also being subject to distance culling – that is, they won’t exist if a player is more than n units away from the entity.

Enabling the variable might help with game performance (FPS), and should help with bandwidth concerns as well as ‘texture loss’/’city bug’ streaming issues caused by loading larger amounts of custom addon player vehicles than usual.

Cons are that player-occupied vehicles will not be known to client scripts unless another player is nearby enough. This might affect ‘teleport into player’s vehicle’ scripts, for instance.

onesync_forceMigration true

This one forces any entity which has not received any clone sync updates for over x seconds to be migrated to any other nearby player. While it doesn’t fix the underlying cause for ‘ghost vehicle/ped’ migration failures, it should make them less painful and not require kicking the alleged owner anymore.

There should not be too many cons, but it’s disabled by default lest it crash players when it’s triggered, or worse.

Useful snippet: getting the top left of the minimap in screen coordinates

SetScriptGfxAlign(string.byte('L'), string.byte('B'))
local minimapTopX, minimapTopY = GetScriptGfxPosition(-0.0045, 0.002 + (-0.188888))
ResetScriptGfxAlign()

local w, h = GetActiveScreenResolution()

return { w * minimapTopX, h * minimapTopY }

This could be useful if you want to align things to NUI or other screen-space draws, and works on pretty much any aspect ratio. The magic numbers here are the minimap position from common:/data/ui/frontend.xml, as seen below:

<data name="minimap"           alignX="L"	alignY="B"	posX="-0.0045"		posY="0.002"		sizeX="0.150"		sizeY="0.188888" /> <!-- WARNING! Feed MUST match posX and PosY -->

This will likely also work for aligning to other positions, assuming you set the right sizes.

Improving script stack traces

In the past, the Citizen framework scripting runtime would print really messy stack traces such as.. this:

Stack traces across runtimes.

This stack trace goes on for around 3-4 times the displayed length, and is generally.. a big mess. You can tell that some resource code is running, like @stest3/sv.lua:20, f.js:8 and @stest1/sv.lua, but everything else that is shown is just noise.

Starting at today’s Windows server (and client, on canary) releases, we’re doing something new and exciting: stack boundary stitching. This is the third attempt at implementing cross-runtime stack tracing, and the first one that actually had the possibility to make it out the door.

To compare, look at the new stack trace output:

A stitched stack.

You’ll see that we’re filtering out only user code, and we’re also showing each error only once: every error down the stack due to failed reference calls simply gets hidden. In addition to that, we now format C# functions using a ‘friendly’ library with support for providing nice method names.

An API and more functionality for stack traces (showing a call boundary, JS source map support) might be offered in the future – for now this is a feature which is generally runtime-internal, and is supported across runtimes: Mono/C#, V8/JS and Lua.

It’d also be really nice if you’d provide some more suggestions regarding developer experience improvements you’d love to see. We think this one will help a lot diagnosing script issues at a glance, however!

GET_ACTIVE_PLAYERS: the replacement for player loops

Just a quick hint: when writing new client-side scripts in Lua/JS (C# already has the Players list doing exactly this), you can loop through players by using the GET_ACTIVE_PLAYERS native. See below for an example of before/after:

Before…

for i = 0, 255 do
    if NetworkIsPlayerActive(i) then
        local ped = GetPlayerPed(i)
        -- do stuff
    end
end

After…

for _, player in ipairs(GetActivePlayers()) do
    local ped = GetPlayerPed(player)
    -- do stuff
end

This should be a bit more performant than 256 native invocations, even if no player slot exists, and it also doesn’t differ based on server-side sync technology slot count – all players with an active ped should be returned by this native.

Adaptive Cards in deferrals

We’ll be writing some more documentation on this soon, but the latest server version (combined with the canary client at this time – no prod yet!) supports presenting Adaptive Cards to clients during deferrals.

Here’s a quick code sample:

AddEventHandler('playerConnecting', function(name, skr, d)
    d.defer()

    Wait(50)

    -- badly serialized JSON in a string, from the Adaptive Cards designer
    d.presentCard([==[{"type":"AdaptiveCard","body":[{"type":"TextBlock","size":"ExtraLarge","weight":"Bolder","text":"Server password?!"},{"type":"TextBlock","text":"That's right, motherfucker! You have to enter a goddamn PASSWORD to connect to this server...","wrap":true},{"type":"Input.Text","id":"password","title":"","placeholder":"better enter one now"},{"type":"Image","url":"http://images.amcnetworks.com/ifccenter.com/wp-content/uploads/2019/04/pulpfic_1280.jpg","altText":""},{"type":"ActionSet","actions":[{"type":"Action.Submit","title":"Sure..."},{"type":"Action.ShowCard","title":"YOU WISH!!!!!!","card":{"type":"AdaptiveCard","body":[{"type":"Image","url":"https://i.imgur.com/YjMR0E6.jpg","altText":""}],"$schema":"http://adaptivecards.io/schemas/adaptive-card.json"}}]}],"$schema":"http://adaptivecards.io/schemas/adaptive-card.json","version":"1.0"}]==],
        function(data, rawData)
            -- you can chain cards, this is just the example adaptive card in the designer
            d.presentCard([==[{"type":"AdaptiveCard","body":[{"type":"Container","items":[{"type":"TextBlock","size":"Medium","weight":"Bolder","text":"Publish Adaptive Card schema"},{"type":"ColumnSet","columns":[{"type":"Column","items":[{"type":"Image","style":"Person","url":"https://pbs.twimg.com/profile_images/3647943215/d7f12830b3c17a5a9e4afcc370e3a37e_400x400.jpeg","size":"Small"}],"width":"auto"},{"type":"Column","items":[{"type":"TextBlock","weight":"Bolder","text":"Matt Hidinger","wrap":true},{"type":"TextBlock","spacing":"None","text":"Created {{DATE(2017-02-14T06:08:39Z,SHORT)}}","isSubtle":true,"wrap":true}],"width":"stretch"}]}]},{"type":"Container","items":[{"type":"TextBlock","text":"Now that we have defined the main rules and features of the format, we need to produce a schema and publish it to GitHub. The schema will be the starting point of our reference documentation.","wrap":true},{"type":"FactSet","facts":[{"title":"Board:","value":"Adaptive Card"},{"title":"List:","value":"Backlog"},{"title":"Assigned to:","value":"Matt Hidinger"},{"title":"Due date:","value":"Not set"}]}]}],"actions":[{"type":"Action.ShowCard","title":"Set due date","card":{"type":"AdaptiveCard","body":[{"type":"Input.Date","id":"dueDate"},{"type":"Input.Text","id":"comment","placeholder":"Add a comment","isMultiline":true}],"actions":[{"type":"Action.Submit","title":"OK","url":"http://adaptivecards.io"}],"$schema":"http://adaptivecards.io/schemas/adaptive-card.json"}},{"type":"Action.Submit","title":"View","url":"http://adaptivecards.io"}],"$schema":"http://adaptivecards.io/schemas/adaptive-card.json","version":"1.0"}]==],
                function(_, rawData2)
                    -- normally you'd check the password, also submit button IDs are sent as submitId
                    -- we're lazy so just reject everyone and tell them the password
                    d.done('you suck actually for entering the password: ' .. data.password .. ' and data like ' .. rawData2)
                end)
        end)
end)

Lua support for compile-time Jenkins hashes

In GTA/RAGE games, you’ll often encounter native functions either taking or returning a Jenkins one-at-a-time hash, like the ones returned by GET_HASH_KEY.

Traditionally, code would have had to either hardcode hashes (like -1044093321), or use GET_HASH_KEY directly (like GetHashKey("a_m_y_skater_01")). This would have lead to either less readability and maintainability, or having to wait for a slow native invocation.

Starting in releases of FiveM/FXServer shipping within the next week, CfxLua has gained support for a new string literal: the backtick string — ``. Now, you’ll have an alternative to the former two, that will still be readable, and will be replaced with a hash during Lua compilation time: `a_m_y_skater_01`.

Let’s look at a real-world example:

CreateThread(function()
    while true do
        Wait(0)

        local f = 0
        local ped = PlayerPedId()

        for i = 1, 10000 do
            if GetEntityModel(ped) == `a_m_y_skater_01` then
                f = f + 1
            end
        end

        print(f)
    end
end)

This Lua code runs at around 18 milliseconds per tick on our testing system. The old-style equivalent would’ve run a lot worse – replacing the `a_m_y_skater_01` part with GetHashKey('a_m_y_skater_01') results in a tick time of 43 milliseconds on the same system – more than twice as slow!

Our work on supporting emoji in Scaleform GFx

In a recent commit to the canary branch, we’ve added support for embedding standard emoji in nearly all cases where text is rendered by Scaleform GFx. This post details a bit of the effort that went on behind this, and where future changes may be headed.

Where does GTA use Scaleform GFx?

You might already know the use of Scaleform GFx (hereafter, GFx) for playing back some of the built-in Flash movies that R* created, to look more like the stock game interface. Some of you might even have been so adventurous as to use an ancient version of Flash that can generate bytecode supported by GFx v3 (v4 refactored a lot of components, including AS3 support and a new renderer, but V does not use this version) to create your own movie clips.

However, GFx also offers a functionality for drawing ‘immediate-mode’ text, which is used as backing API for a lot of text-related script commands (including the ever-popular BEGIN_TEXT_COMMAND_DISPLAY_TEXT, core to many servers’ custom HUD draws), to allow these commands to use the same font support that actual Flash movies played in GFx have.

On emoji

For rendering emoji in a text rendering engine, a few approaches can be used, and there have been a few competing systems in the past. The ‘lazy’ solution would be to use a simple emoji font that uses TTF glyphs (like Microsoft’s “Segoe UI Emoji”) and render the black-and-white fallback glyphs into a normal SWF font. This works fine, but you don’t have nice-looking emoji like that – they’re just black-and-white outlines.

Windows uses this emoji style in a few places.

These really are hard to distinguish – the potato is already hard to recognize, but at this size, who can see the second one is a pineapple?

A simple approach to get these emoji rendered with colors would be to layer these glyphs in different colors at the same position, but of course the SWF DefineFont3 specification (which is the latest one existent – GFx defines a custom “compressed font” as well, which is even more trivial) doesn’t allow more than one shape to be placed for a specific glyph, let alone specifying colors for these non-existent multiple shapes.

Now, it was thought of to simply enable HTML text formatting for all text fields and immediate text draws, since Flash (and, to a limited extent, GFx) supports a HTML-like formatting system: tags like <B>, <I>, <IMG>, <FONT> and a few other basic ones are existent and usable. Especially <IMG> is interesting: this is already an approach used on the web to support emoji for systems that either do not support emoji or have an incomplete set of emoji.

However, as it later turned out, GFx’ <IMG> tag did not support vector images (Flash ‘sprites’, which are actually an independent movie clip which can have multiple frames and recursively-placed sprites, shapes and other objects, are the closest it gets to multi-layer vector images), and looking at the GFx SDK documentation showed almost no hope as the renderer was deemed too complex to even integrate this functionality in, and more tauntingly, a header file indicated that ‘sprite image’ support was implemented at one point.

SVGs in Flash

First, a quick digression: we in fact were looking for a way to convert Twemoji’s SVG images to SWF easily for use in this case, but most conversion software was either even more outdated than Flash itself, just simply didn’t work, produced incorrect results, or all of the former facts. Eventually, however, we found out that the Adobe Flex compiler had a fairly decent import functionality for SVG images, however it also added a lot of arbitrary AS3 bytecode, which would have massively tripped up the GFx Flash converter/loader.

All those useless scripts…

However, from back when Flash was relevant, a lot of parties defined an XML representation for SWF which allowed us to quickly process the (by then 130 MB) XML file to remove all these definition tags using a standard streaming XML reader/writer, and then import the now-stripped XML data into a functionally equivalent SWF file, only containing the sprite and shape definitions we needed.

Except – as above, there was no sprite support.

The solution in the end

Eventually, we ended up looking at the GFx SDK for the right version – we’d been looking at version 4 before, and GTA uses version 3, and eventually stumbled on a much simpler renderer backend. However, sprites were still not implemented, but some leftover code was found that pointed the way towards handling sprites in text rendering, and an effort started to end up supporting creating temporary instances of sprites (parented to a dummy movie clip – as GFx expects a movie clip), and eventually drawing them inline in the text formatting renderer.

An early attempt at getting formatting of a few random emoji.

Eventually, we got closer…

… much better formatting and replacement.

… and added support for DrawTextManager and not only text fields, since for some reason GFx had nearly the same code duplicated twice…

vMenu, with all strings replaced for testing

… and using a few different emoji showed not all of them were square…

Not square.

… which eventually was fixed!

Aspect ratio! Spacing!

And, finally, even weird emoji combiners worked:

That’s a SHRUG + EMOJI MODIFIER FITZPATRICK TYPE-1-2 + MALE SIGN, with a crazy amount of combiners in between.

Which shows – it finally works!

The road to shipping

At this time, there are a few issues left preventing this from being put on the production channel – HTML support ‘everywhere’ led to some code to escape non-HTML text input, but this also incorrectly escaped some actual text input; and we’re still working on changing around some emoji alignment positions to make larger text not get layouted incorrectly just because an emoji is present in the input.

Once that’s done, you’ll probably be able to use emoji everywhere – both in NUI, as well as in anything rendered by GFx.

As to international font support – this is a completely unrelated issue, but rest assured that we are planning on adding more scripts to the built-in font libraries – however, we can’t easily (or at all) implement certain so-called ‘complex scripts’ like Arabic, since a) GFx has no support for these, at all and b) implementing a script processor for these is a project by itself, and not something that can be easily done by people without years upon years of experience in Unicode script processing.