JSON Introduction

JSON, JavaScript Object Natation, is a lightweight data interchange format that is ideal for server interaction with JavaScript.

In ordinary Web applications, developers often struggle with XML parsing, either server-side generation or processing of XML, or client-side parsing of XML with JavaScript, often resulting in complex code and very low development efficiency. In fact, for most Web applications, many AJAX applications even return pieces of HTML directly to build dynamic Web pages. Returning HTML fragments greatly reduces the complexity of the system compared to returning XML and parsing it, but lacks some flexibility.

JSON provides an alternative data interchange format for Web application developers. Like XML, JSON is a plain text-based data format. Because JSON is inherently intended for JavaScript, the JSON data format is very simple. You can use JSON to transfer a simple String, Number, Boolean, an array, or a complex Object object.

The structure of JSON is based on the following two points

  • Collection of name/value pairs understood in different languages as object, record, structure, dictionary, hash table, keyed list, etc.
  • ordered lists of values are understood as arrays in most languages

JSON represents JavaScript objects as a specific string. If a string of this form is assigned to any JavaScript variable, the variable becomes an object reference, and the object is constructed from the string.

Suppose here that we need to create a User object with the following properties.

  • User ID
  • User name
  • Website

You can use the following JSON form to represent the User object.

1

{"UserID":100, "Name":"biaodianfu", "website":"https://www.biaodianfu.com"};

Then if this string is given to a JavaScript variable, then either property of the object can be used directly.

1
2
3
4

<script>
    var User = {"UserID": 100, "Name": "biaodianfu", "Website": "https://www.biaodianfu.com"};
    alert(User.Name);
</script>

The actual use may be a little more complicated, for example, if we define a more detailed structure for Name, so that it has EnglishName and ChineseName.

1

{"UserID":100, "Name":{"EnglishName":"biaodianfu","ChineseName":"标点符"}, "Website":"https://www.biaofianfu.com"};

Full code.

1
2
3
4
5
6
7
8

<script>
    var User = {
        "UserID": 100,
        "Name": {"EnglishName": "biaodianfu", "ChineseName": "标点符"},
        "Website": "https://www.biaofianfu.com"
    };
    alert(User.Name.ChineseName);
</script>

If a page needs a list of users, not just a single user information, then you need to create a list of users array. The following code demonstrates the use of JSON to define this list of users.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

<script>
    var UserList = [
        {
            "UserID": 100,
            "Name": {"EnglishName": "biaodianfu", "ChineseName": "标点符"},
            "Website": "https://www.biaofianfu.com "
        },
        {
            "UserID": 101,
            "Name": {"EnglishName": "biaodianfu", "ChineseName": "标点符"},
            "Website": "https://www.biaofianfu.com "
        },
        {
            "UserID": 102,
            "Name": {"EnglishName": "biaodianfu", "ChineseName": "标点符"},
            "Website": "https://www.biaofianfu.com "
        },
    ];
    alert(UserList[0].Name.ChineseName);
</script>

In addition to using the “.” reference attribute, the following statement can be used.

1
2

alert(UserList[0]["Name"]["ChineseName"]);
alert(UserList[0].Name["ChineseName"]);

Some rules of JSON.

  • An object is a collection of attribute and value pairs. An object starts with “{” and ends with “}”. Each attribute name and value is prompted with “:” and separated by “,” between attributes.
  • An array is an ordered collection of values. An array starts at “[” and ends at “]”, and values are separated by “,”.
  • Values can be quoted strings, numbers, true, false, null, or objects or arrays. All these structures can be nested.
  • Strings and numbers are defined in much the same way as in C or Java.

Some advantages of JSON.

  • JSON provides an excellent object-oriented method for caching metadata to clients.
  • JSON helps separate validation data from logic.
  • JSON helps provide the essence of Ajax for Web applications

Difference between JSON and Python dictionary

JSON and Python dict is very similar, are stored in the form of key-value data, and json, dict can also be very convenient through dumps, loads for mutual conversion of the format.

Since they are all in key-value format, why do they need format conversion?

  • JSON (JavaScript Object Notation): json is a data format, which is a pure string. It can be parsed into Python’s dict or other forms.
  • dict (dictionary): A dictionary is a complete data structure, an implementation of the data structure HashTable, a set of schemes that encapsulates everything from storage to extraction. It uses a built-in hash function to plan the storage location of the key corresponding to the value, so as to obtain a faster data reading speed.

The difference between JSON and dict.

  • json’s key can only be a string, python’s dict can be any hashable object.
  • json’s key can be ordered and repeated, python’s dict’s key can’t be repeated.
  • There is a default value undefined for json’s key, and no default value for dict.
  • The value of json can only be a string, float, boolean, or null, or an array or object of them.
  • json can be accessed as [], or as . The value of a dict can only be accessed by the subscript [].
  • The string of json is forced to be double-quoted, and the string of dict can be single-quoted or double-quoted.
  • Only arrays are available in json, dict can nest tuples.
  • Chinese in json must be unicode encoding, such as “hello” in json should be “\u4f60\u597d”.
  • The data type of json is string (str), and the data type of dictionary is dictionary (dict).
  • json defines boolean and null values: true, false, null.
  • Python defines boolean and null values: True, False, None.

Python standard library JSON module

JSON data can be coded and decoded in Python using the json module, which contains two functions.

JSON string to Python dictionary

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# Import the module
import json

# String with JSON format
data_JSON = """
{
    "size": "Medium",
    "price": 15.67,
    "toppings": ["Mushrooms", "Extra Cheese", "Pepperoni", "Basil"],
    "client": {
        "name": "Jane Doe",
        "phone": "455-344-234",
        "email": "janedoe@email.com"
    }
}
"""

# Convert JSON string to dictionary
data_dict = json.loads(data_JSON)
print(type(data_dict))
print(data_dict["size"])
print(data_dict["price"])
print(data_dict["toppings"])
print(data_dict["client"])

Output content.

1
2
3
4
5

<class 'dict'>
Medium
15.67
['Mushrooms', 'Extra Cheese', 'Pepperoni', 'Basil']
{'name': 'Jane Doe', 'phone': '455-344-234', 'email': 'janedoe@email.com'}

Data type conversion in the JSON to Python dictionary process.

Python dictionary to JSON string

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# Import the module
import json

# Python Dictionary
client = {
    "name": "Nora",
    "age": 56,
    "id": "45355",
    "eye_color": "green",
    "wears_glasses": False
}

# Get a JSON formatted string
client_JSON = json.dumps(client)
print(type(client_JSON))
print(client_JSON)

Output content.

1
2

<class 'str'>
{"name": "Nora", "age": 56, "id": "45355", "eye_color": "green", "wears_glasses": false}

Data type conversion for Python dictionary to JSON.

Python JSON data beautification

1
2

client_JSON = json.dumps(client, indent=4)
print(client_JSON)

Output content.

1
2
3
4
5
6
7

{
    "name": "Nora",
    "age": 56,
    "id": "45355",
    "eye_color": "green",
    "wears_glasses": false
}

JSON file reading and writing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Open the orders.json file
with open("orders.json") as file:
    # Load its content and make a new dictionary
    data = json.load(file)

    # Delete the "client" key-value pair from each order
    for order in data["orders"]:
        del order["client"]

# Open (or create) an orders_new.json file 
# and store the new version of the data.
with open("orders_new.json", 'w') as file:
    json.dump(data, file)

Difference between json and simplejson

We see many projects using the simplejson module instead of the json module in the standard library. Why use these alternatives instead of the alternatives in the standard library?

The json module in the Python standard library is actually simplejson, except that it is updated more slowly than the standalone simplejson. Both are similar in functionality, but simplejson is a bit better in performance.

It is recommended to use it like this.

1
2
3
4

try:
    import simplejson as json
except ImportError:
    import json

Other JSON parsing tools

  • python-rapidjson: RapidJSON is an efficient C JSON parser and generator open source by Tencent. It is currently packaged as a Python library.
  • ultrajson: json parsing library written in C
  • pysimdjson: Python binding for the simdjson project, which is a SIMD-accelerated JSON parser. If no SIMD directive is available, a fallback parser is used so that pysimdjson is safe to use anywhere.
  • orjson: orjson is written in Rust language, no local Rust environment he will prompt you need Rust environment, installation is more troublesome, performance is said to be very fast.
  • marshmallow: marshmallow is an ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, to and from native Python datatypes.

Python and JSONP

JSONP (JSON with Padding) is a JSON “usage pattern” that allows web pages to request data from other web domains.

Due to the same-origin policy, web pages located at server1.example.com generally cannot communicate with servers that are not server1.example.com, with the exception of the <script> element in HTML. Using this open strategy of the <script> element, web pages can get JSON information dynamically generated from other sources, and this usage pattern is known as JSONP. The information captured with JSONP is not JSON, but arbitrary JavaScript, which is executed with a JavaScript translator instead of being parsed with a JSON parser.

To understand how this pattern works, imagine that there is a URL that returns a JSON file, and the Javascript program can use XMLHttpRequest to ask for information from this URL. Suppose our URL is http://server2.example.com/RetrieveUser?UserId=xxx. Suppose Ming’s UserId is 1234, and when the browser passes Ming’s UserId through the URL, that is, it crawls http://server2.example.com/ RetrieveUser?UserId=1234, we get.

1

{"Name": "小明", "Id": 1823, "Rank": 7}

This JSON information may be generated dynamically based on the query parameters passed to the URL.

In this case, it is conceivable to set the src attribute of the <script> element to a URL that returns the JSON, which means that it is possible to grab the JSON from the HTML page through the script element.

However, a JSON file is not a JavaScript program. In the JSONP usage model, the URL is a dynamically generated JSON wrapped in a function call, which is the “padding” or “prepending” of JSONP. " or “prefix” in JSONP.

By convention, the browser provides the name of the callback function as part of the named query parameters in the request sent to the server, e.g.

1

<script type="text/javascript" src="http://server2.example.com/RetrieveUser?UserId=1234&jsonp=parseResponse"></script>

The server fills the callback function (parseResponse) with JSON data before passing it to the browser. The response the browser gets is no longer a mere narrative of information but a script. In this case, the browser gets the following.

1

parseResponse({"Name": "Cheeso", "Id": 1823, "Rank": 7})

While the padding (pre-padding) is “usually” a callback function defined in the context of browser execution, it can also be a variable assignment, an if narrative, or other JavaScript narrative. the response to a JSONP request (i.e., a request using the JSONP schema) is not JSON and is not treated as such JSON parsed - the return can be any operator, and need not even be JSON at all, although the padding part is still conventionally a small snippet of JavaScript that triggers a function call that acts on the JSON-formatted material.

Alternatively, a typical JSONP is a solution that wraps an existing JSON API with a function request to achieve cross-domain access. In order to initiate a JSONP request (or rather, to use this pattern), you need a script element. Therefore, the browser must add (or reuse) a new <script> element with the required src value into the HTML DOM for each JSONP request - or “inject” the element. Because of this, JSON is said to be a way to “allow users to bypass same-origin policies by injecting script elements”.

Using the script tag of the remote site allows the remote site to inject any content into the site. If the remote site has a JavaScript injection vulnerability, the original site will be affected as well. There is an ongoing project to define a so-called JSON-P strict security subset that would allow browsers to force the processing of requests with a MIME class of “application/json-p”. If the response cannot be parsed as strictly JSON-P, the browser can throw an error or ignore the entire response.

Rough JSONP deployments are vulnerable to cross-site forgery requests (CSRF/XSRF). Because HTML <script> tags do not adhere to the same-origin policy in browsers, malicious pages can request and obtain JSON information belonging to other sites. When a user is logged into that other site, this situation allows the malicious site to manipulate the JSON data in the context of the malicious site, potentially compromising the user’s password or other sensitive information.

This is only a problem if the JSON data contains confidential information that should not be leaked to third parties, and the server relies only on the browser’s same-origin policy to block unusual requests. If the server determines the proprietary nature of the request itself and only outputs the information if the request is normal, then there is no problem. A cookie alone is not sufficient to determine that a request is legitimate, which makes it vulnerable to cross-site forgery of requests.

JSONP client-side implementation

We know that even if the code in the cross-domain js file (which of course means in compliance with the web script security policy), the web page can be executed unconditionally.

The remote server remoteserver.com has a remote.js file in the root directory with the following code.

1

alert('我是远程文件');

There is a jsonp.html page code under the local server localserver.com as follows.

1

<script type="text/javascript" src="http://remoteserver.com/remote.js"></script>

The remote.js file code is as follows.

1

localHandler({"result":"我是远程js带来的数据"});

After running and checking the result, the page successfully pops up a window showing that the local function was called successfully by the remote js across the domain and also received the data brought by the remote js. I’m very happy that the purpose of cross-domain remote data acquisition has basically been achieved, but another problem has arisen, how do I let the remote js know what the name of the local function it should call is? After all, the server of jsonp has to face many service objects, and each of these service objects has a different local function, right? Let’s move on to the next step.

Smart developers can easily think, as long as the server provides js script is dynamically generated on the line, so the caller can pass a parameter to tell the server “I want to call XXX function of the js code, please return to me”, so the server can be in accordance with the needs of the client to generate js script and respond to the client’s request.

Look at the code of the jsonp.html page.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

<script type="text/javascript">
    // 得到航班信息查询结果后的回调函数
    var flightHandler = function (data) {
        alert('你查询的航班结果是：票价 ' + data.price + ' 元，' + '余票 ' + data.tickets + ' 张。');
    };
    // 提供jsonp服务的url地址（不管是什么类型的地址，最终生成的返回值都是一段javascript代码）
    var url = "http://flightQuery.com/jsonp/flightResult.aspx?code=CA1998&callback=flightHandler";
    // 创建script标签，设置其属性
    var script = document.createElement('script');
    script.setAttribute('src', url);
    // 把script标签加入head，此时调用开始
    document.getElementsByTagName('head')[0].appendChild(script);
</script>

This time the code change is relatively large, no longer directly write the remote js file to death, but the code to achieve dynamic query, and this is the core part of the jsonp client implementation, the focus in this case also lies in how to complete the whole process of jsonp calls.

We see that the call url passed a code parameter, telling the server that I want to check the CA1998 flight information, and the callback parameter tells the server that my local callback function is called flightHandler, so please pass the query results into this function to call.

OK, the server is very smart, the page called flightResult.aspx generates a piece of code like this to jsonp.html (the server-side implementation is not demonstrated here, it has nothing to do with your choice of language, in the end it is a spliced string).

1
2
3
4
5

flightHandler({
        "code": "CA1998",
        "price": 1780,
        "tickets": 5
    });

We see that what is passed to the flightHandler function is a json, which describes the basic information about the flight. Run the page, a pop-up window appears successfully, and the whole process of jsonp execution is completed successfully!

Up to this point, I believe you have been able to understand the principle of jsonp client implementation, right? All that remains is to encapsulate the code so that it can interact with the user interface and be called multiple times and repeatedly.

jQuery implementation of JSONP

We still use the same example as above for the flight information query, assuming that the jsonp results are returned unchanged.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

<script type="text/javascript" src=jquery.min.js"></script>
<script type="text/javascript">
    jQuery(document).ready(function () {

        $.ajax({
            type: "get",
            async: false,
            url: "http://flightQuery.com/jsonp/flightResult.aspx?code=CA1998",
            dataType: "jsonp",
            jsonp: "callback",//传递给请求处理程序或页面的，用以获得jsonp回调函数名的参数名(一般默认为:callback)
            jsonpCallback: "flightHandler",//自定义的jsonp回调函数名称，默认为jQuery自动生成的随机函数名，也可以写"?"，jQuery会自动为你处理数据
            success: function (json) {
                alert('您查询到航班信息：票价： ' + json.price + ' 元，余票： ' + json.tickets + ' 张。');
            },
            error: function () {
                alert('fail');
            }
        });
    });
</script>

Why didn’t I write the function flightHandler this time? And it worked! That’s where jQuery comes in. jquery automatically generates a callback function for you when handling ajax of type jsonp and takes the data out for the success property method to call.

Similarities and differences between ajax and jsonp

ajax and jsonp are two technologies that “look” very similar in the way they are called, and the purpose is the same, both are requesting a url and then processing the data returned by the server, so frameworks such as jquery and ext encapsulate jsonp as a form of ajax; but ajax and jsonp are essentially different things.

But ajax and jsonp are actually essentially different things. ajax’s core is to get non-page content via XmlHttpRequest, while jsonp’s core is to dynamically add <script> tags to call the js script provided by the server.

So, in fact, the difference between ajax and jsonp is not whether it is cross-domain or not. ajax can achieve cross-domain through the server-side proxy as well, and jsonp itself does not exclude the acquisition of data from the same domain.

And that is, jsonp is a way or non-compulsory protocol, as ajax, it does not have to use json format to pass data, if you want, string can, but this is not conducive to the use of jsonp to provide public services.

In short, jsonp is not a special case of ajax, even if jquery and other giants to jsonp encapsulated into ajax, can not change this!

Python with JSONP

Python generates JSONP data.

1
2
3
4
5
6

def dumps_jsonp(request):
    funcname = request.GET.get('callback')
    # print(funcname)
    content = '%s(%s)' % (funcname, data())
    # print(content)
    return HttpResponse(content)

Python reads JSONP data.

1
2
3
4
5
6
7
8

import re
import json

def loads_jsonp(_jsonp):
    try:
        return json.loads(re.match(".*?({.*}).*",_jsonp,re.S).group(1))
    except:
    raise ValueError('Invalid Input')