#29 Internet of Things with ESP8266/Arduino IDE #1: Calling RESTful JSON Services

#29 Internet of Things with ESP8266/Arduino IDE #1: Calling RESTful JSON Services


Greetings YouTubers! Here is the guy with a Swiss accent again. The ESP8266 Is a great product. It adds internet connectivity to a quite fast processor, and with the newer boards like the ESP 12 we also have plenty of memory and pins available. And with the Arduino IDE we have a well known programming environment and lots of libraries. But what is lacking for a true internet of things platform? We need a software framework with the following characteristics: Simple connections to Web services; IOT devices have to be connected to the world; Simple interface to smartphone; IOT devices have to be simple and often without keyboard or display. This is why they have to be able to get the information for Wi-Fi credentials and other parameters directly from the smartphone. They need a sleep mode to extend battery life. We all know that, ESP8266 need lots of Energy and cannot work long time with batteries. New software has to be loaded remotely over-the-air. If such devices are deployed they physically cannot be reached anymore because they are built in or have no programming connectors. Future users would never want to deal with FTDI or other programming devices. I start now the building of such a software platform and will show you my progress Today I start with the first point And I hope I will be able to reach the last point in the next weeks To prove the concept I decided to build a device which connects to an official web service in Switzerland, which announced is all departures of public transportation. It then compares the actual time with the departure time and acoustically warns the traveler if the vehicle arrives. This is very useful because it works completely automatic and also takes into account the time the traveler needs to walk or run to the next station. And it takes also into account if the vehicle is delayed, if the web service provides real-time information. Of course you could get this information using an app on your smartphone But this is very inconvenient because it needs many steps till you get the result, and you have to repeat it several times to get the newest information. My device will stand in our kitchen and will be started if I decide to take the next bus. To build the device I started with understanding web services. in contrary to Normal HTML pages, Web services provide information in machine readable form. Popular formats used by these services are XML and JSON. With IOT devices we have to preserve memory space. This is why I prefer JSON over XML interfaces. Therefore in this video, I will concentrate on this technology. JSON or Javascript Object Notation is an open standard format that uses human readable text to transmit data objects in a standardized format. JSON messages are provided by so-called RESTful Services which communicate over HTTP and use verbs like get, post, put, delete, etc. Because my IOT device only wants to read information, we will concentrate on the GET work. As a first step we have to find the web service which provides us with the needed information. This is typically done by a Google search or by going to the site https://www.programmableweb.com/ Let’s assume you found an API which delivers the information you need. Then it is good to read its description to understand the syntax of the call and If you know the syntax the best next step is to try it out manually. To do this I installed an app from the Google Chrome web store. You find the link below in the comments. The syntax of the Swiss Transportation Api is on this slide. It consists of three parts: an address, the two cities where the journey starts and ends, and optional a statement to restrict the number of fields which are returned from the web service. This part is very important because it reduces the effort to find the right information on our small IOT device. Let’s test this service. As already said, I need the GET message. I write the call in the respective line and hit send. I get now three parts of information. First I get a status message and second I get the header information. The third part is the information I am interested in. It displays the next departure from Basel to Zurich. This message has only 67 bytes. if I switch to raw I see the string which I have to decode later in my program on my ESP8266. But let’s play a little with the api. If I remove the optional field restriction, the message is much bigger. It has now 1764 bytes, and it would be nearly impossible to handle it on a small IOT device. Here we learn an important lesson. Keep the big work in the internet and not on your small device The return message is exactly as you see it here. The first line will be the date line and the last line will be the connections line. Our program just has to check if the status of the message is okay. It is status 200; and then read through all lines to find the sub-string “departure”. The numbers of desire are then in a constant distance from this keyword. Hours minutes and seconds of the next departure. Let’s now look into the code. We have [to] read and decode the response. Bigger computers usually read the whole message in one block and decode or parse it afterwards. For small devices we read line after line and search for the keyword “departure”, here passed to the subroutine as the variable lookupString. If we found it in a particular line we extract the three variables: hours, minutes, and seconds, using string arithmetic. At the end we can return the next departure time to the calling code But how is the request message looking? It consists of three lines. The first is similar to the one we used in the manual scenario, but without the host name. The second line adds the host information, and the third instructs the server if we want to keep the connection alive or close it after the request. The respective program code is straightforward. Do not forget the carriage return and line feed characters. Here I want to call your attention to a particularity of the ESP8266. The WiFi traffic is handled in parallel to your normal program execution and you do not feel it because the ESP firmware handles it without intervention by stealing some time of your program. if you do loops this stealing can be prohibited and the ESP crashes. If this happens you have to enhance waiting loops with a short delay. Never wait in a while (1==1) loop. The ESP will crash immediately and produce a dump. Only if you enhance your loop with a small delay(1), your ESP will run without any problems. In this video we were able to discover a web service providing a RESTful API. Check it out manually. Program a call from the ESP8266. and decode the JSON formatted messages. Now we are ready to move on with our IOT framework. In the next video I will show you how to connect your ESP to your smartphone or PC to enter the needed credentials for the Wi-Fi Network and other data. Here you see the current status. Every seconds the ESP8266 queries the web service and starts to peep if the time to leave for the bus arrives. I hope this video was useful or at least interesting for you. Bye!

Danny Hutson

49 thoughts on “#29 Internet of Things with ESP8266/Arduino IDE #1: Calling RESTful JSON Services

  1. Hallo Andy, schicke Lösung. Hast Du schon diese Library für JSON untersucht? https://github.com/bblanchon/ArduinoJson/blob/master/README.md

  2. i downloaded your project from GitHub, it is really awesome, but i am having problems because the project don't have the librarie credentials.h, and the other libraries, for example, helpers.h and global.h, i think have some mistakes.
    do you can help me to solve this problem?

    i really apreciate that

    thansk for your share

  3. Interfaced arduino uno with esp8266-01 Wi-Fi module, 16*2 LCD …now I have a webpage where it has two text boxes and submit button.So now when I type something on text box and click on submit button ,the text should get displayed on LCD . I tried for led toggle from internet example .it is working . at commands everything is working . I am new to server and web concept. I tried get method with HTML in webpage.but could not succeed. Some people said try php. Any idea.

  4. I am enjoying your videos. I want to play, but… There are too many choices! Andreas, help the confused noobies out.
    How about doing an "Idiot's guide to ESP8266 models, and what to look for to make sure you get something good for cheap"?
    I've spent a couple days looking at listings from FastTech, Banggood, DealExtreme, AlieExpress, etc. and sheesh there's a lot out there. Are those companies reliable, are those usable items, or is that "no-name junk"?
    Some sites advertise a 12F…. I don't think that's a typo, because I see it all over – but 12F isn't listed on the ESP8266 wiki.
    Can you explain what the meaning and significance is if it says NodeMCU, "Witty", LUA, WeMos, or stuff like that? What's a ESPduino? A "Witty cloud development" board? Is it worth getting a board that advertises "1M flash" or is that still too small for OTA? Argh!
    I've seen a couple "beta boards" and dev boards for less than $10 that include built-in stuff like temp/humidity sensors, RGB LED, relays, switches… it would sound like this would let you be up and experimenting immediately – but there's no documentation.
    Color me frustrated. I just want a quick and simple way to play around with web-enabled micro controllers.

  5. Please don't query web services every second. It gives a time when you query it; you could store that time to a variable, and loop (using sufficient yield() instructions) for a second, decrementing the time each loop, and end up with the same result.

    Even the NTP queries could be removed entirely by adding a simple RTC Module…

    Querying the same information every second wastes your bandwidth (and the servers), and with enough concurrent devices, could overload the web service. Try to limit repeated connections as much as possible.

  6. Hallo Andreas,

    erstmals vielen Dank! Super Videos die du da machst, habe schon sehr viel gelernt 😉

    Ich habe einen ESP8266 programmiert mit der Arduino IDE, damit er von einem DHT22 Temperatur und Luftfeuchtigkeit an Thingspeak sendet. Funktioniert alles soweit recht gut. Nur ist der ESP auch als Accesspoint sichtbar und das möchte ich nicht. Wie kann ich das deaktivieren? Ist das in einem speziellen Register abgelegt?

    Mfg

    Stefan (auch mit Swiss Accent 😉

  7. I have LoLin NodeMcu v3 board and it doesn't respod to commands, like AT. And when I restart it with the button it shows this:

    ets Jan 8 2013,rst cause:2, boot mode:(3,6)

    load 0x4010f000, len 1384, room 16
    tail 8
    chksum 0x2d
    csum 0x2d
    v09f0c112
    ~ld

    What should I do?

  8. Really REALLY need to be editing out that OBNOXIOUS beeping sound… I've come to expect much more from your videos. Thank you for the efforts regardless!

  9. Sir,
    I have watched your previous videos which help me a lot. But I am facing a problem for using NODE MCU 12 E OUTPUT pin, that is I am unable to use GPIO19(D0), GPIO0(D3),GPIO3(D9) as output pin. They are not giving any OUTPUT(HIGH/LOW) . is it not possible to use them as output pins? which pins can be used for input/output?

    any help in this regards will be highly appreciated.

    thank you.

    DEBASISH BANERJEE

  10. Hi Andreas, you can use this lib #include <ArduinoJson.h> for decoding encondig JSON strings it will simplify it it.
    it works under ESP , https://github.com/bblanchon/ArduinoJson , I used it for enconding data over a serial stream and it seems to work very well..

  11. Dear Friend With the Swiss Accent,

    I truly enjoy watching your Tutorials. You must enjoy doing them, as I know it must be quite time consuming and involve a lot of work.

    I have watched others, but find yours to be among the absolute best. So many treat the subject matter as common knowledge, often skipping over some of the more important information. Then go on to talk about how they have overclocked the board or system and are able to send enormous amounts of useless data from nonexistent apps. they never intend to put together to all parts of the globe.

    Thank you for taking the time to share your wealth of knowledge. It is truly helpful, and easy to understand. And, your accent is no trouble at all to understand, but does help in remembering wonderful people from the world over can work together to make good things happen.

    Robert Blaylock

  12. Hi Andreas
    Greetings from Denmark and thanks for some great videos.
    You are such an expert in ESP8266, so maybe you can guide me in the right direction 😀

    I'm building a IR remote for my "homeautomation system", and have a ready to go solution with IR connected to a UNO.
    Now I want to control it with a esp8266f, to get it wireless, so it can talk with the the homeautomation server. But when i'm compiling the code I get a interrupt error..of course (as the esp8266 don't have this interrupt)
    It's a fairly simple code for a UNO, so the question is
    1. do I find another esp8266 type
    2. CAN it be done with a esp8266f at all
    3. do I find another IR library
    4. do I use the UNO as IR sender and talk with the esp8266f via serial communication
    5. do I change the source code in the IR library (no sure if it can be done)

    I use Ken Shirriff's awesome IRremote library: https://github.com/shirriff/Arduino-IRremote
    And it do not support the esp8266!

    Here is his Hardware specifications:

    Board/CPU Send Pin Timers
    ATtiny84 6 1
    ATtiny85 1 TINY0
    ATmega8 9 1
    Atmega32u4 5, 9, 13 1, 3, 4
    ATmega48, ATmega88, ATmega168, ATmega328 3, 9 1, 2
    ATmega1284 13, 14, 6 1, 2, 3
    ATmega164, ATmega324, ATmega644 13, 14 1, 2
    ATmega8535 ATmega16, ATmega32 13 1
    ATmega64, ATmega128 13 1
    ATmega1280, ATmega2560 5, 6, 9, 11, 46 1, 2, 3, 4, 5
    Teensy 1.0 17 1
    Teensy 2.0 9, 10, 14 1, 3, 4_HS
    Teensy++ 1.0 / 2.0 1, 16, 25 1, 2, 3
    Teensy 3.0 / 3.1 5 CMT
    Teensy-LC 16 TPM1
    The table above lists the currently supported

    My idea was nr 4, but again I think it's a lot of work to do, and when thinking of the esp8266f 's memory and cpu power.
    It should definitely do it all and more, so I think it's a overkill.

    Some of the hardware in the list I know and some I don't. But it dont list the esp8266…. 🙁
    Any ideas would be great.

    So thanks in advance.
    Tim M.

    —————————-
    Compile with Uno:

    Sketch uses 3,724 bytes (11%) of program storage space. Maximum is 32,256 bytes.
    Global variables use 526 bytes (25%) of dynamic memory, leaving 1,522 bytes for local variables. Maximum is 2,048 bytes.
    —————————-
    Compile with ESP8266 :

    #include <avr/interrupt.h>
    compilation terminated.
    Using library Arduino-IRremote-master at version 2.2.1 in folder: C:UsersTestDocumentsArduinolibrariesArduino-IRremote
    exit status 1
    Error compiling for board Generic ESP8266 Module.

  13. https://github.com/SensorsIot/Internet-of-Things-with-ESP8266 not there. Get 404 error. Can you post the code again?

  14. Thank you for your excellent Tutorials, we have several ESP8266 and DA14580 BLE modules, LoRa module with FCC, CE certification, I wonder is it possible that you help me test these modules?
    Thank you for sharing valuable knowledge with us.
    It is really helpful for wirelss tech fans.

  15. i want to know how to get json data and make as a source json after send request elasticsearch

    #include<ESP8266WiFi.h>
    #include<ArduinoJson.h>

    //const char* ssid "Dialog 4G"
    //const char* "[email protected]"

    const char* ssid = "Crystal Martin Visitors";
    const char* password = "[email protected]";

    const char* serverAddress="tracer-stg-lb1.southcentralus.cloudapp.azure.com";
    const int serverPort=9200;
    void setup() {
    Serial.begin(9600);
    connectWIFI();

    }

    void connectWIFI(){
    Serial.println("Try to connect wifi..");
    WiFi.begin(ssid,password);
    while(WiFi.status()!=WL_CONNECTED){
    delay(100);
    Serial.print(".");
    }
    Serial.println("Connected");
    Serial.print("IP Address-");
    Serial.println(WiFi.localIP());

    }

    void myClient(){
    WiFiClient client;
    while(!client.connect(serverAddress,serverPort)){
    Serial.println("Connection failed…");
    return;
    }

    //Serial.println("Connected..");

    String url = "/esp/data/1 ";
    client.print(String("GET ") + url + "HTTP/1.1rn" +
    "Host: tracer-stg-lb1.southcentralus.cloudapp.azure.com:9200rn" +
    "Connection: closern" +
    "Content-Type: application/jsonrn" +
    "Cache-Control: no-cachern" +
    "rn");

    while(!client.available()){

    Serial.println("Try…");

    }
    client.readStringUntil('n');
    client.readStringUntil('n');
    client.readStringUntil('n');
    client.readStringUntil('n');
    client.readStringUntil('n');
    client.readStringUntil('n');
    client.readStringUntil('n');

    String str = client.readString();
    if (str.substring(77)){

    }
    Serial.println(str);
    StaticJsonBuffer<200> jsonBuffer;
    JsonObject& root = jsonBuffer.parseObject(str);

    if (!root.success())
    {
    Serial.println("parseObject() failed");
    return;
    }

    ////
    //// Step 3: Retrieve the values
    ////
    const char* factory = root["factory_id"];
    const char* line = root["line_id"];
    const int id = root["device_id"];
    const char* time = root["cycle_time"];
    //
    //
    Serial.println(factory);
    Serial.println(line);
    Serial.println(id);
    Serial.println(time);

    //
    //
    // Serial.println(response);
    // Serial.print("Closing connection. ");

    }

    void loop() {
    myClient();

    }

    this is my code but did'n get separate each value

  16. Nice video! Two additional hints that were not mentioned but might be helpfull:
    1. Some webtechnologies allow you to choose in which format you want the HTTP Reply by adding .xml or .json at the end of the uri.
    2. Use yield() instead of delay(1). This allows the scheduler to switch context (and do WiFi stuff) without the break.

  17. Marvellous Andreas, always steered clear of Web Service API's, but will now dive into the IOT and see what I can find! Many thanks, keep up the grand work….

  18. I know this video is a bit old, however I´m hoping you will get this message anyway. I recently bought a bunch of stuff from china, Nevertheless I think I might have messed my self up trying to be cheep. I bought a few of these things and similar {links below}. Who knows maybe I´m not the only person to make this mistake (assuming it was a mistake).
    Link 1: https://www.aliexpress.com/item/Adapter-for-Nodemcu-Lua-USB-32M-ESP8266-ESP-12E-F-CP2102-Internet-Wifi-Module-DC-Power/32837635356.html
    Link 2: https://www.aliexpress.com/item/ESP8266-ESP12E-ESP12F-WIFI-Internet-Development-Board-CH340G-Blue-ESP-12E-ESP-12F-Adapter-Compatible-For/32841023309.html
    Are these just boards that I am supposed to solder Esp12f (or similar) boards onto? Or are these boards full fledged NodeMcu modules already and I´m just over thinking things? I think I went to cheep for my own good and messed myself up.
    Anyway I love you channel. You really help fools like myself out a lot.Please keep up the great work. BTW If I do need to add another board to these should I get the 12f? Also what is the latest and greatest micro-controller It seems to change daily?

  19. Plz Help esp8266 Package Downlod Error Dekha Raha hai Plz Link This Click see screnshoot picture https://scontent.fccu1-1.fna.fbcdn.net/v/t1.0-9/33965456_1045005068986125_8764574593684865024_n.jpg?_nc_cat=0&oh=2cb21ebc5a52054cdb77bb2238ef6030&oe=5B8025A0

  20. Ok.. for future reference.. beeps that loud, repeated more than once or twice, are AnNoYiNg!!!! And then it got faster!.. eesh…

  21. Thank you for sharing with us all this information , unfortunately I'm a beginner that starting to explore it. I'm trying to follow your ESP8266 videos and I'm missing one basic part which is what are the type of models we have ? which one to buy? how to define and connect it to the arduino IDE some basic stuff, I'm sorry if i missed such and can u please direct me to such information?

  22. Dear friend, I hope you continue helping people the way you do on these videos, I just found your videos in 2019 and they really help me with some projects I have for my home city! Thank you!

  23. Hello Andreas I hope you can help me. I'm using a ESP8266 an my problem is that yesterday my code worked perfectly, then made some changes on some conditions and now I still can upload both codes, and create a server in which I can interact but neither the new nor the old code turns the relay on and won't send info to the serial monitor.

  24. My legendary engineer, a greeting from Italy.
    About our beloved ESP8266, how about taking a look at this link?
    I am convinced it can be very interesting, especially with regards to the "microscopic" 01 and 01-S which, personally, I adore and which you showed us as to "BOOST UP".

    https://bbs.espressif.com/viewtopic.php?f=7&t=889

Leave a Reply

Your email address will not be published. Required fields are marked *