Converting a String unicode representation of emoji in SQL to the actual Emoji in Dart (or in PHP)

I have spent the whole day and night on this. Many answers from many people but none of which works for my case. Sorry for the noob question but I literally have spent whole yesterday until early morning for this.

So I got an old system to migrate to a new one. Essentially from SQL server to Firebase. With a new structure, all new uid and the client wants the unicode data from old SQL server to be converted to the real emojis.

So it goes from SQL (in old server) => PHP (in old server to fetch the SQL data) => Flutter (in my laptop) => Firebase (Firestore).

In the SQL server there are strings like this for example:
Spring... fall...ud83dudc90...flowers!

Then the PHP code that retrieved from the SQL database return it like this:

// Create Connection
$conn = new mysqli($servername, $username, $password, $dbname);
mysqli_set_charset($conn,'utf8mb4');
    
header("Access-Control-Allow-Origin: *");

$result = $conn->query($sql);
    
if($result->num_rows > 0) {
    $db_data = array();
    while($row = $result->fetch_assoc()) {
        $db_data[] = $row;
    }

    // Send back the complete records as a json
    echo json_encode($db_data, JSON_UNESCAPED_UNICODE);
}else{
    http_response_code(405);
    echo "Query error";
}
$result->free();
$conn->close();
return;

In Dart (Flutter Web) I got the data like this:

final response = await http.get(urlRequest);

final rawRows = jsonDecode(response.body) as List<dynamic>;
  
rawRows.forEach((map) => print(map['description']));

This prints out: Spring... fall...ud83dudc90...flowers!

What I want is Spring... fall...💐...flowers!

I did a hardcoded print test (i.e: print('Spring... fall...ud83dudc90...flowers!'); and it gives exactly the result I want (i.e: It turns ud83dudc90 part of the string to 💐)

I have tried sending it as is to Firebase, it still doesn’t recognize ud83dudc90 as emojis. It was stored in Firebase like this Spring... fall...ud83dudc90...flowers!. Which is totally understandable, I guess we need to convert it first.

I also have tried using converter in the PHP side like this: (thanks to Pedro Lobito)

function unicodeString($str, $encoding=null) { 
    if (is_null($encoding)) $encoding = ini_get('mbstring.internal_encoding');
    return preg_replace_callback('/\\u([0-9a-fA-F]{4})/u', create_function('$match', 'return mb_convert_encoding(pack("H*", $match[1]), '.var_export($encoding, true).', "UTF-16BE");'), $str);
}

Then the $db_data[] = $row; part in the PHP changed to $db_data[] = unicodeString($row);

But this solution only works for some emojis but not for all emojis, such as the case of 💐 above. It shows Spring... fall...?...flowers! instead.

So now I want to try to convert it in Dart instead since I’m more familiar with it. Unless someone could help to solve it in PHP as well.

Answer

Finally after working a couple of days I worked on my own solution.

So what I did was to use RegEx to detect the sequence in the String and replace them with emojis.

1. Declare Regular expressions for detecting the unicodes (in my case the unicode is something like (uXXX):

// To detect a single unicode
static const regEx1 = '\\u([0-9a-fA-F]{4})'; 

// To detect a 2-bytes unicode
static const regEx2 = '$regEx1$regEx1';

2. Create a method to detect each RegEx. Then when a pattern is found, replace it with jsonDecode with added quotes in there. Like this jsonDecode('"${match.group(0)}"');.

The code:

static String _regExEmojiUnicode(String text, String regEx) {
  final regexCheck = RegExp(regEx, caseSensitive: false);
  String newText = '';
  int _lastEndText = 0;
  int _lastEndNewText = 0;

  regexCheck.allMatches(text).forEach((match) {
    final start = match.start;
    final end = match.end;

    final String replacement = jsonDecode('"${match.group(0)}"');
      
    String startString;
    newText == ''
        ? startString = '${text.substring(0, start)}$replacement'
        : startString =
            '${newText.substring(0, _lastEndNewText)}${text.substring(_lastEndText, start)}$replacement';

    _lastEndNewText = startString.length;
    _lastEndText = end;

    newText =
        '$startString${text.substring(end)}';
      
  });

  if(newText == '') newText = text;

  return newText;
}

3. Create a method to check for the different cases of emojis:

static String replaceEmoji(String text) {
  String newText = text;

  // Checking for 2-bytes and single bytes emojis
  if(newText.contains('\u'))
    newText = _regExEmojiUnicode(newText, regEx2);
  if(newText.contains('\u'))
    newText = _regExEmojiUnicode(newText, regEx1);

  return newText;
}

Then this works!!

I did this in haste, so if there is any improvements can be made to the RegEx patterns or anywhere else in the code, I’m open to any suggestions.

Thank you