SA-MP Forums

Go Back   SA-MP Forums > SA-MP > Bug Reports

Reply
 
Thread Tools Display Modes
Old 17/06/2018, 11:53 AM   #1
[HLF]Southclaw
High-roller
 
[HLF]Southclaw's Avatar
 
Join Date: Apr 2009
Location: England
Posts: 4,952
Reputation: 1548
Default PSA: OnFilterScript* Events Not Thread Safe

Hello SA:MP community, this post is a small warning to those of you who make heavy use of filterscripts on your servers. Just moments ago, a bug was discovered in the filterscript system that can cause race conditions.

What Is Affected

This issue occurs when filterscripts are loaded either
  • From the console (confirmed)
  • Via RCON remotely (confirmed)

If you load and unload filterscripts using these methods as part of your live production gamemode and these filterscripts execute any functions with global side effects, you are at risk of unexpected crashes due to undefined behaviour caused by race conditions.

Here is a list of affected callbacks from Y_Less:
  • OnFilterScriptInit - "loadfs", "reloadfs"
  • OnFilterScriptExit - "unloadfs", "reloadfs", "exit"
  • OnPlayerDisconnect - "kick"
  • OnRconCommand - anything non-standard
  • OnGameModeExit - "exit", "gmx"

What Isn't Affected

From what we can tell
  • SendRconCommand (unless it is called from the affected callbacks)
  • In-game /rcon commands

What Can Be Done?

Stop using loadfs and unloadfs via the console. Currently there is no easy fix for this, however you shouldn't be doing this on live production servers anyway.

If you must remotely load and unload filterscripts, here are some ways to remotely call code in a gamemode from which you can make use of SendRconCommand.
  • IRC - Use Incognito's IRC plugin and create loadfs/unloadfs commands
  • Discord - Use maddinat0r's Discord plugin and create loadfs/unloadfs commands
  • WebSockets - Use my Requests plugin and connect with a webapp or mobile app

---

The details:

Console input is handled like most command-line applications: a second thread runs and captures text that comes in via standard input. These commands are then turned into actions such as `gmx` restarts the mode, `exit` closes the server and `loadfs`/`unloadfs` will load and unload filterscripts, etc.

However, the loading and unloading of filterscripts happens inside the console thread rather than the actual main server thread. So if you call basically any native function or plugin function that has side effects, you run the risk of causing a data race in the server process because there's a change you could be calling a native that affects the same area of memory than a native currently being called in the main thread (either the gamemode or another filterscript).

Illidan deserves credit for discovering this, Yashas, Y_Less and I did a bit of digging and confirmed it. If we discover any more issues related to this, this thread will be updated.

It would be great to see this resolved in a future update as this causes unexpected crashes that are very difficult to trace to one particular source. I don't know the internals of SA:MP but I'm sure some locks and queues could resolve this issue quickly.
__________________
Tools:

Plugins:

Links:


Last edited by [HLF]Southclaw; 19/06/2018 at 03:57 PM.
[HLF]Southclaw is offline   Reply With Quote
Old 17/06/2018, 12:27 PM   #2
IS4
Banned
 
Join Date: Apr 2018
Posts: 15
Reputation: 66
Default Re: PSA: OnFilterScript* Events Not Thread Safe

From what it seems, every command that can be executed from the input thread or remote RCON is not thread-safe. I am using PawnPlus' thread_id (implemented by std::this_thread::get_id) to check the source thread:
Code:
#include <a_samp>
#include <PawnPlus>
native print_s(AmxString:string) = print;

public OnFilterScriptInit()
{
    print("Init:");
    print_s(thread_id_s());
}

public OnPlayerDisconnect(playerid, reason)
{
    print("Disconnect:");
    print_s(thread_id_s());
}

public OnRconCommand(cmd[])
{
    print("RCON:");
    print_s(thread_id_s());
}
The output corresponds with the assumptions:
Code:
  Loading filterscript 'Test1.amx'...
Init:
54216
  Loaded 4 filterscripts.


RCON (In-Game): Player [[KSA]Illidan5_cz] sent command: testcmd
RCON:
54216
Unknown command or variable:
  testcmd


RCON admin connected to server.
RCON:
180044
Unknown command or variable:
  testcmd


kick 0
[KSA]Illidan5_cz <#0 - 127.0.0.1> has been kicked.
Disconnect:
179736


reloadfs Test1
  Filterscript 'Test1.amx' unloaded.
Init:
179736
  Filterscript 'Test1.amx' loaded.
The main thread is identified as 54216 (Test1 is loaded from server.cfg). When I send an RCON command in-game, it is handled by the RCON RPC synchronously, so no problems there. However, even commands that are used more often than loadfs, like kick, can execute code in parallel with a script, if they get executed via external RCON (using the query which is executed in another thread).

Example of parallel code execution using kick:
Code:
#include <a_samp>

stock delay(ticks)
{
    new end = GetTickCount()+ticks;
    while(GetTickCount() < end) {}
}

public OnFilterScriptInit()
{
    for(new i = 0;; i++)
    {
        printf("%d", i);
        delay(1000);
    }
}

public OnPlayerDisconnect(playerid, reason)
{
    printf("Disconnecting %d", playerid);
}
Code:
RCON (In-Game): Player [[KSA]Illidan5_cz] sent command: loadfs Test2
0
1
2
3
Console input: kick 0
[KSA]Illidan5_cz <#0 - 127.0.0.1> has been kicked.
Disconnecting 0
4
5
When using crashdetect, this can even crash the server, since the plugin doesn't expect running code concurrently.

Edit: Since remote RCON is handled by the server query mechanism, basically any query command (ping, players, rules etc.) can be potentially thread-unsafe. Moreover, the "BAD RCON ATTEMPT BY" message is produced by logprintf which is not thread-safe.

Last edited by IS4; 17/06/2018 at 01:19 PM.
IS4 is offline   Reply With Quote
Old 17/06/2018, 12:37 PM   #3
[HLF]Southclaw
High-roller
 
[HLF]Southclaw's Avatar
 
Join Date: Apr 2009
Location: England
Posts: 4,952
Reputation: 1548
Default Re: PSA: OnFilterScript* Events Not Thread Safe

This also solves a forgotten mystery I once ran in to. My gamemode had an infinite loop (I think it was caused by random(0) or valstr, can't remember) but during the loop I could still load and unload filterscripts and the code in OnFilterScriptInit still ran - so I still saw messages printed, it just never did anything else after that.
__________________
Tools:

Plugins:

Links:

[HLF]Southclaw is offline   Reply With Quote
Old 18/06/2018, 12:00 AM   #4
Freaksken
Gangsta
 
Freaksken's Avatar
 
Join Date: Mar 2009
Location: Belgium
Posts: 619
Reputation: 287
Default Re: PSA: OnFilterScript* Events Not Thread Safe

This might explain this:
__________________
Always keep in mind that a lot of people are active on this forum in their spare time.
They are sacrificing time they could easily spend on things they would rather do, to help you instead.


[Include] FCNPC A.I. - [Tutorial] Speedometer - [WIP] Single-player-like co-op mission - [Other] List of all my work
Freaksken is offline   Reply With Quote
Old 18/06/2018, 02:53 AM   #5
IstuntmanI
High-roller
 
IstuntmanI's Avatar
 
Join Date: Mar 2009
Location: Romania
Posts: 1,862
Reputation: 373
Default Re: PSA: OnFilterScript* Events Not Thread Safe

Quote:
Originally Posted by Freaksken View Post
This might explain this:
Exactly what I thought after I have read the description of the problem.

Interesting discovery, this is probably why I always felt like executing RCON commands in the command line wasn't safe at all. I also noticed weird behavior when executing something in the server command line, especially when loading and unloading filterscripts. Weird that nobody else brought this problem in discussion until now.

I guess that now I have one more reason to be against the use of filterscripts, everything is better and easier when using just a gamemode.
__________________


IstuntmanI is offline   Reply With Quote
Old 18/06/2018, 05:49 AM   #6
Sew_Sumi
Banned
 
Join Date: Jun 2008
Location: Azerbaijan
Posts: 6,469
Reputation: 525
Default Re: PSA: OnFilterScript* Events Not Thread Safe

Holy crap, and here I was thinking of a gamemode script that would load and unload the filterscripts in the past few months...
Sew_Sumi is offline   Reply With Quote
Old 18/06/2018, 07:20 AM   #7
Kar
High-roller
 
Kar's Avatar
 
Join Date: May 2010
Location: Black Asylum
Posts: 3,184
Reputation: 506
Default Re: PSA: OnFilterScript* Events Not Thread Safe

2018 baii, the world is real.
Kar is offline   Reply With Quote
Old 18/06/2018, 09:45 AM   #8
Spmn
Gangsta
 
Join Date: Jun 2015
Location: Romania
Posts: 540
Reputation: 131
Default Re: PSA: OnFilterScript* Events Not Thread Safe

Quote:
Originally Posted by Sew_Sumi View Post
Holy crap, and here I was thinking of a gamemode script that would load and unload the filterscripts in the past few months...
... Which is still safe.
Spmn is offline   Reply With Quote
Old 18/06/2018, 09:58 AM   #9
[HLF]Southclaw
High-roller
 
[HLF]Southclaw's Avatar
 
Join Date: Apr 2009
Location: England
Posts: 4,952
Reputation: 1548
Default Re: PSA: OnFilterScript* Events Not Thread Safe

Quote:
Originally Posted by Sew_Sumi View Post
Holy crap, and here I was thinking of a gamemode script that would load and unload the filterscripts in the past few months...
If you did that pragmatically via SendRconCommand, you'll be fine. It's only console and RCON that's affected.
__________________
Tools:

Plugins:

Links:

[HLF]Southclaw is offline   Reply With Quote
Old 18/06/2018, 11:30 AM   #10
Y_Less
Beta Tester
 
Y_Less's Avatar
 
Join Date: Jun 2008
Location: 629 - git.io/Y
Posts: 15,127
Reputation: 3170
Default Re: PSA: OnFilterScript* Events Not Thread Safe

List of affected callbacks, and the console commands used to trigger them:

OnFilterScriptInit - "loadfs", "reloadfs"
OnFilterScriptExit - "unloadfs", "reloadfs", "exit"
OnPlayerDisconnect - "kick"
OnRconCommand - anything non-standard
OnGameModeExit - "exit", "gmx"

Basically, anything triggered by typing in the console, and anything called from those callbacks.
Y_Less is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Events! AyanAliKhan Server Support 1 15/11/2016 12:02 PM
events StR_MaRy Scripting Help 3 16/04/2016 08:19 AM
Halloween events Eagle1928 Everything and Nothing 1 25/10/2014 05:04 PM
[Map] MY FIRST MAP FOR EVENTS by Georgia tujituji1 Maps 14 16/01/2011 10:53 AM
[HELP] Events andershh Help Archive 4 17/07/2010 02:28 AM


All times are GMT. The time now is 09:48 AM.


Powered by vBulletin® Version 3.8.6
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.