currentWeather from OpenWeatherMap shows no data on ESPaper everything else is fine

Question

I switched from the WeatherUnderground version to the new OpenWeatherMap code (downloaded 10th October 2018 from GitHub). ESPaper is connected to my WiFi, data is updating, forecast is showing plausible values but no icon for current weather and no description text for current weather. Temp is showing 0. I tried two different OPEN_WEATHER_MAP_LOCATION_IDs both show same effect (with different data for forecast).

Is the struct OpenWeatherMapCurrentData still uptodate (in file ./Arduino/libraries/ESP8266_Weather_Station/src/OpenWeatherMapCurrent.h)?

Thanks for any help.

Ingo

Answer 1

You can check the OWM response for 'current weather' like this:

http://api.openweathermap.org/data/2.5/weather?q=London,UK&APPID=your_API_key&mode=json&units=metric&cnt=1

Add your apI key and see what the JSON response is, like this, check for 'e.g. 'Clouds':

{"coord":{"lon":-0.13,"lat":51.51},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04d"}],"base":"stations","main":{"temp":19.49,"pressure":1003,"humidity":68,"temp_min":18,"temp_max":21},"visibility":10000,"wind":{"speed":7.2,"deg":170},"clouds":{"all":75},"dt":1539262200,"sys":{"type":1,"id":5093,"message":0.0057,"country":"GB","sunrise":1539238704,"sunset":1539278097},"id":2643743,"name":"London","cod":200}

Comments

Thanks for this hint. I got:

{"coord":{"lon":13.39,"lat":52.52},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01d"}],"base":"stations","main":{"temp":24,"pressure":1017,"humidity":36,"temp_min":24,"temp_max":24},"visibility":10000,"wind":{"speed":5.1,"deg":120},"clouds":{"all":0},"dt":1539267600,"sys":{"type":1,"id":4892,"message":0.0029,"country":"DE","sunrise":1539235537,"sunset":1539274771},"id":2950159,"name":"Berlin","cod":200}

I received even better values for "main", "description", "icon" and "temp" right now here in Berlin :-)

I also changed the query from ...weather?q=Berlin,DE&APPID=... to ...weather?id=2950159&APPID=...

My API key and the location id I use seem ok.

---

I tried the Arduino Demo project found on the Thingpulse GitHub page here (https://github.com/ThingPulse/esp8266-weather-station/tree/master/examples/OpenWeatherMapCurrentDemo). My result in the monitor window are:
...
Next Loop-Step: 51406:
Getting url: http://api.openweathermap.org/data/2.5/weather?id=2950159&appid=HERE_IS_MY_CORRECT_OWM_APP_ID&units=metric&lang=de
[HTTP] GET...
[HTTP] GET... code: 200
------------------------------------
lon: 0.000000
lat: 0.000000
weatherId: 0
main:
description:
icon:
iconMeteoCon:
temp: -159465800836912022475758179315756302336.000000
pressure: 61438
...
Seems all wrong! When I copy and paste the URL into my browser the results is correct.

---

I guess we should have a mode in the streaming JSON parser that dumps all data to serial. What we're seeing above is the "processed" data i.e. no way to tell for sure that the data the device got from OWM is really correct.

---

Ideally yes a serial output of the JSON stream would help solve these issues. I tried Berlin this afternoon and all was as expected for the forecast, so I can't see why it failed, other than an OWM server issue or over-use of the API and the request was being temporarily blocked.

---

My forecast data looks good just the currentWeather data is wrong. I don't think that it is overuse of the API, I use the free account but my update interval is set to 20 minutes. Do you get the same results for currentWeather or is it just on my setup?

---

I just tried Berlin,DE and got this response:

{"coord":{"lon":13.39,"lat":52.52},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}],"base":"stations","main":{"temp":14.98,"pressure":1018,"humidity":63,"temp_min":14,"temp_max":16},"visibility":10000,"wind":{"speed":3.1,"deg":120},"clouds":{"all":0},"dt":1539294600,"sys":{"type":1,"id":4892,"message":0.003,"country":"DE","sunrise":1539235571,"sunset":1539274728},"id":2950159,"name":"Berlin","cod":200}

And it is entirely normal and works for me, has your code been modified?

---

Overuse is always an option but a) 60 calls per hour is fairly generous b) forecast data would be affected as well I assume.

I'm starting to suspect there might (still) be a bug in parsing the current data from OWM. One was fixed with weather station lib 1.6.2. We haven't heard from the OP what version he uses but we have heard from other users about that "current temperature is 0"-issue. We never managed to reproduce it ourselves, though.

---

Have you catered for up to 3 reports in:
"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}]

My location frequently gets 3, so 'Clouds' was (this morning) Mist, Rain and Fog in 3 separate array sub-scripts. OWM report across a region where there can be variations in weather e.g. one side of a large town and the other.

weather":[{"id":803,"main":"Rain","description":"Light Rain","icon":"04n"},{"id":803,"main":"Mist","description":"Misty","icon":"04n"},{"id":803,"main":"Fog","description":"Fog","icon":"04n"}]

---

Yes and no, we just ignore all but the first weather report: https://github.com/ThingPulse/esp8266-weather-station/blob/master/src/OpenWeatherMapCurrent.cpp#L113

---

When I compare my results that I got from copying the URL to my browser and the results of @G6EJD the only difference I found is that my temp value is always an integer not a float as in your result. I changed from metric to imperial and again got only an integer result.
Maybe this gives a hint to the problem?

---

This is what's used as the API call:

"http://api.openweathermap.org/data/2.5/weather?" + locationParameter + "&appid=" + appId + "&units=" + units + "&lang=" + language;

So identical and the code correctly converts temperature to float, then pressure and  humidity to int so nothing wrong there.

It's a mystery.

If you can add a diagnostic print in here:
https://github.com/ThingPulse/esp8266-weather-station/blob/master/src/OpenWeatherMapCurrent.cpp#L113

after line 138

Serial.println(temp); // which should display current temperature.

---

An API call with units=imperial

{"coord":{"lon":13.39,"lat":52.52},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}],"base":"stations","main":{"temp":68.88,"pressure":1023,"humidity":46,"temp_min":68,"temp_max":69.8},"visibility":10000,"wind":{"speed":4.7,"deg":140},"clouds":{"all":0},"dt":1539363000,"sys":{"type":1,"id":4892,"message":0.0022,"country":"DE","sunrise":1539322054,"sunset":1539361022},"id":2950159,"name":"Berlin","cod":200}

An API call with units=imperial
{"coord":{"lon":13.39,"lat":52.52},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}],"base":"stations","main":{"temp":20.49,"pressure":1023,"humidity":46,"temp_min":20,"temp_max":21},"visibility":10000,"wind":{"speed":2.1,"deg":140},"clouds":{"all":0},"dt":1539363000,"sys":{"type":1,"id":4892,"message":0.0023,"country":"DE","sunrise":1539322054,"sunset":1539361022},"id":2950159,"name":"Berlin","cod":200}

Calls used:

http://api.openweathermap.org/data/2.5/weather?q=Berlin,DE&APPID=your_key_here&mode=json&units=imperial&cnt=1

http://api.openweathermap.org/data/2.5/weather?q=Berlin,DE&APPID=your_key_here&mode=json&units=metric&cnt=1

Both results show a floating point temperature. It could have been the temperature at your reading time was a whole number.

---

I added some diagnostic println in the OpenWeatherMapForecast.cpp at line 63 and following:

...
  Serial.printf("[HTTP] GET... code: %d\n", httpCode);
  Serial.println("*******************");
  if(httpCode > 0) {
    Serial.println("httpCode > 0");

    WiFiClient * client = http.getStreamPtr();

    while(client->connected()) {
        Serial.println("connected");
        while((size = client->available()) > 0) {
              if ((millis() - lost_do) > lostTest) {
                  Serial.println ("lost in client with a timeout");
                  client->stop();
                  ESP.restart();
            }
          Serial.println("no timeout");
          c = client->read();
          if (c == '{' || c == '[') {
            isBody = true;
          }
          if (isBody) {
            parser.parse(c);
          }
          // give WiFi and TCP/IP libraries a chance to handle pending events
          yield();
        }
      }
  }
  this->data = nullptr;
...
My monitor shows this:
...
About to call OpenWeatherMap to fetch station's current data...
Getting url: http://api.openweathermap.org/data/2.5/weather?id=2950159&appid=a623f384a032e2b0520b2492b6d09ad8&units=metric&lang=de
[HTTP] GET...
[HTTP] GET... code: 200
*******************
httpCode > 0
Getting url: http://api.openweathermap.org/data/2.5/forecast?id=2950159&appid=HERE_IS_MY_APP_ID&units=metric&lang=de
[HTTP] GET...
[HTTP] GET... code: 200
...

The OpenWeatherMapCurrent::doUpdate sends the http code but IS NOT CONNECTED.

Does the problem has something to do with the daily forecast not available for the free account?

---

If I click on your supplied links above (with the API key)< I get a normal response:

{"coord":{"lon":13.41,"lat":52.52},"weather":[{"id":800,"main":"Clear","description":"Klarer Himmel","icon":"01d"}],"base":"stations","main":{"temp":22,"pressure":1015,"humidity":37,"temp_min":22,"temp_max":22},"visibility":10000,"wind":{"speed":6.2,"deg":140},"clouds":{"all":0},"dt":1539517800,"sys":{"type":1,"id":4892,"message":0.3885,"country":"DE","sunrise":1539495038,"sunset":1539533578},"id":2950159,"name":"Berlin","cod":200}

So I'm wondering if somehow your code is corrupt, have you tried downloading a fresh copy?

---

Oh, I forgot to remove my app code from both URLs :-)
When I click on the my link I got a correct answer too. The return code in the monitor window says 200 which means: ok. For me it seems that the code to handle the json stream for the CURRENT weather is not working. The almost same code for the FORECAST works fine. The code for the current weather didn't even connect to the current weather stream.
At the end of the OpenWeatherMapCurrent::doUpdate procedure (https://github.com/ThingPulse/esp8266-weather-station/blob/master/src/OpenWeatherMapCurrent.cpp line 83 and 84) a comment says "give the libs time to handle events". This means that before the calling of the next procedure (the forecast procedure) the lib has the time and it seems that this call is gives a correctly initialised client. Maybe it is just that for the first call of the streaming parser there is not enough time and the first call has no proper connection.

---

Heureka!

I tried a delay(10000); but that didn't led to a better result. I inserted a
Serial.println(client->connected());
right after WiFiClient * client = http.getStreamPtr(); and saw a 0 in the console window.
Then I commented out the command     "while(client->connected()) { " and the corresponding "}" and NOW I see the correct result on my display!!!

Please check if this is ok to update your code base.

Thanks for the hints and the discussion!

---

What speed is your ISP service, is it particularly fast? Maybe 80MBit/sec or faster.

---

@David, I'm curious what you're hinting at with your question? Care to explain? I'm on a 100+MBit fiber connection and I never experienced this issue.

So, it looks like we're talking about the following lines of code: https://github.com/ThingPulse/esp8266-weather-station/blob/master/src/OpenWeatherMapCurrent.cpp#L62-L69. Ingo's analysis seems to show that `WiFiClient`'s internal connection state (`connected()`) says the client be disconnected. Yet, reading from the client is ok as it does have data in its buffer available (`client->available() > 0`). If that's indeed the case then this would sound like a bug in Arduino-core to me. I'm a bit puzzled.

---

I am on a 25Mit ISP service line. I found this discussion and it seems to me similar to our problem:
https://github.com/espressif/arduino-esp32/issues/1527

---

... and also this https://github.com/esp8266/Arduino/issues/4227

---

Marcel, I’m using a 80Mb service and can’t replicate the issue, so I concluded this might be a timing issue with perhaps a slower connection. It may be a core issue but no-one seems to have responded to the links just posted to confirm the issue.

Ingo saw no data when there should have been leading me to conclude the data had not yet arrived when the test started (he got 0) and when he removed the test it worked.

So is the code sensitive to connection speed? I don’t know as it’s very difficult for most of us to test. I don’t see any code errors of course, so it hints of a timing issue to me.

---

Those two issues Ingo linked to describe the exact opposite if I got that right i.e. connected() not returning false even though the connection was lost/close.

What I thought about after I sent my previous comment is this; a hypothesis. What if the client "instantly" fills its internal buffer and since the response is so small (just the current weather) it - or the server - closes the connection. So, it's no longer connected but all available data was already fetched.

---

I agree with your hypothesis and I have seen something similar on the ESP8266 when I built some sensor clients that uploaded (temperature, humidity and pressure) to a server, as what I saw was the connection taking place, the server acknowledging the connection but no data was received by the server. I solved this by adding a short 500mS delay at the server end to allow the connection to stay alive just a little longer and I presume enable the data to be processed by both client and server.

As the protocol is HTTP and TCP the connection should be kept alive until the transfer is complete, so this may need client.keepalive which I believe exists in WifiClient.h

so would need a call to client.keepalive(true,time,time,cnt); // enable, idle-time,inteval, time, count e.g. client.keepalive(true,10000,10000,5);
client.connect…

Looks like it has been implemented: https://github.com/esp8266/Arduino/pull/3937

---

I inserted this line "client->keepAlive(10000UL, 10000UL, 0x5);" right after line 65. The project compiled but the esp8266 was then in a reboot loop :-(
Maybe I did it wrong. What could I check?

---

For now it may be better to insert a small delay on line 65, maybe 500mS (delay(500);) and try different values. I think you said you removed the connected test and it worked?

keepalive is enabled by keepAlive(true,100UL, 100UL, 0x1); but the documentation is sparse as to how it operates.

---

Yes, I removed the connected test and the ESPaper is working now.

I just reinserted it back again and tried the keepAlive method. I inserted keepAlive (no boolean first parameter) and I got this reboot loop that I saw in the console window.

---

You could try changing the connected test to this:
while((client->available())) {

---

David, wouldn't that effectively be the same as the inner while loop?

---

Hi Marcel, it might be but to be honest I'm not yet that sure where Ingo made his modifications.

Hi Ingo, could you post your working code i.e. with lines disabled please.

This is an intriguing problem...

---

Just to make the state of my tests clear. My only code changes are in the file OpenWeatherMapCurrent.cpp, this is how the lines 64 and following look like on my system:
...
  if(httpCode > 0) {
    WiFiClient * client = http.getStreamPtr();
//    client->keepAlive(10000UL, 10000UL, 0x5);   // Ingo: not working -> esp rebooting
//    while(client->connected()) {                                // Ingo: works with this test commented out
        while((size = client->available()) > 0) {
              if ((millis() - lost_do) > lostTest) {
                  Serial.println ("lost in client with a timeout");
                  client->stop();
                  ESP.restart();
            }
          c = client->read();
          if (c == '{' || c == '[') {
            isBody = true;
          }
          if (isBody) {
            parser.parse(c);
          }
          // give WiFi and TCP/IP libraries a chance to handle pending events
          yield();
        }
      }
//  }                                                                                      // Ingo: see above
  this->data = nullptr;
...
There is still the test in the code that checks for available chars.

---

I am working on a Mac, file is located on "iCloud Drive: Documents/Arduino/libraries/ESP8266_Weather_Station/src/OpenWeatherMapCurrent.cpp"

---

So for the shorter 'current weather' data returns (small) your getting false for:

while(client.connected())

When it should be true and at that instance it does look like the server dropped the connection, hence the attempt to keepalive the client connection so it can pass the connected test was the logic.

---

Right. The idea of inserting the keepAlive seems good. However I didn't get it working.

---

Checking through the protocol definition I see that 'HTTP/1.1 has keep-alive enabled by default' so what request type is being sent? Probably HTTP/1.1 :)

---

How can I check that?

---

When the HTTP GET is made the protocol type is appended usually like this "GET /") + code + " HTTP/1.1\r\n" …

Buit the actualy GET is in this form:

int httpCode = http.GET();

And I have not found the GET function to check

---

I am not sure, but I guess the file ESP8266HTTPClient.h is used. Located on my Mac Home Directory -> Library⁩ ▸ ⁨Arduino15⁩ ▸ ⁨packages⁩ ▸ ⁨esp8266⁩ ▸ ⁨hardware⁩ ▸ ⁨esp8266⁩ ▸ ⁨2.4.2⁩ ▸ ⁨libraries⁩ ▸ ⁨ESP8266HTTPClient⁩ ▸ ⁨src⁩

---

This is the line that precedes the test for the connection and is post the http.get(); call, so this is where I'm focussing.
If you were to add a line before while.client.connected()

Serial.print("Connected status is: ");serial.println(client.connected());

You should see a true (1) after the forecast and false (0) just after the weather call. But on the weather call variant you might see a value less than 0, that would signify an error.

Try that.

I still think we should try to establish what's different between your ESP to OWM server and say mine, like speed or ping time or what, as that's the only difference assuming the code is near identical.

The MAC platform should be irrelevant, but is it?

---

Hi, code:
...
  // start connection and send HTTP header
  int httpCode = http.GET();
  Serial.printf("[HTTP] GET... code: %d\n", httpCode);
  if(httpCode > 0) {
    WiFiClient * client = http.getStreamPtr();
    Serial.print("OpenWeatherMapCurrent: before client->connected() - status = ");
    Serial.println(client->connected());
    while(client->connected()) {                // Ingo: works with this test commented out
      Serial.print("OpenWeatherMapCurrent: after client->connected() - status = ");
      Serial.println(client->connected());
        while((size = client->available()) > 0) {
              if ((millis() - lost_do) > lostTest) {
                  Serial.println ("lost in client with a timeout");
                  client->stop();
                  ESP.restart();
            }
          c = client->read();
          if (c == '{' || c == '[') {
            isBody = true;
          }
          if (isBody) {
            parser.parse(c);
          }
          // give WiFi and TCP/IP libraries a chance to handle pending events
          yield();
        }
      }
  }                                             // Ingo: see above
  this->data = nullptr;
...
Output:
...
Getting url: http://api.openweathermap.org/data/2.5/weather?id=2950159&appid=xxx&units=metric&lang=de
[HTTP] GET...
[HTTP] GET... code: 200
OpenWeatherMapCurrent: before client->connected() - status = 0
Getting url: http://api.openweathermap.org/data/2.5/forecast?id=2950159&appid=xxx&units=metric&lang=de
[HTTP] GET...
[HTTP] GET... code: 200
OpenWeatherMapForecast: before client->connected() - status = 1
OpenWeatherMapForecast: after client->connected() - status = 1
start document
...
Of course temp = 0 on display.
I am also sure that using the MAC platform is irrelevant. Just added that info to let you know that I am using the right files.

---

Oh,of course the same code in the other file.

---

Ping http://api.openweathermap.org gives these results:
29 packets transmitted, 29 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 29.325/29.799/30.175/0.238 ms

---

Very interesting confirmation of what's going on (wrong) and I'm beginning to conclude the ESP core is correct as it may well be the case that the connection was made, data transferred and then connection dropped all before it gets to reading the buffer contents, therefore the outer loop test:

while(client->connected()) {   }

Is not needed, as it then tests for buffer content and reads the buffer anyway. This could mean we have to change the de-facto way network connections are processed.

This could be an artefact of the streaming parser and small packet sizes,  as clearly it handles large amounts of data that would usefully exceed any buffer size.

---

I subscribed to the free OpenWeatherMap service 10th October 2018. I setup 2 different api keys, one for the ESPaper device, one for use in my openHAB installation. 2 days ago I tried the Arduino code with the other api key with the same results.

---

Ping statistics for http://api.openweathermap.org / 178.62.207.82:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 18ms, Maximum = 20ms, Average = 19ms

So yours is a slower connection (~ +50%), but is that the reason for this anomalous behaviour?

---

I doubt that it has something to do with the connection speed or ping time.
I can't find any hardware revision number on my ESPaper board. The ESP module says ESP-WROOM 02. I bought it in December 2017 from Thingpulse.

---

Signal strength on the ESPaper as displayed is very good: 80%. I use a FritzBox 7590 as my router here at home. ESPaper is connected to a repeater 1750E that is connected to the router via mesh connectivity (as the company AVM defines this).

---

I’m thinking network hardware/performance not your ESP. Extenders can in my experience be slow devices and you have recorded slower connection/transit times than most, noting you seem to have a problem that others don’t. The question remains why?

---

I pinged again the OWM server this morning. I measured from my computer connected via ethernet cable: 34ms. Then computer via wifi: 37ms. Then I unplugged the repeater: 37ms. The wifi strength was down to 44%, but still ok.

---

I think I might not be the only one with this problem, Marcel said that some other people reported a 0-temp-issue too. I also don't think that my problem has something to do with the quality of my connection, which I think is above average (I agree that it is 50% slower than yours!). David, you think that the data is transferred and buffered and then the connection is already closed before the next line of code tested an open connection. I agree with that. One difference between your processing of your json data and mine is a difference in length of the json return string because of different languages and different current weather conditions. Maybe we are working just at the length of the buffer. Maybe OWM is treating not all members equal. I can imagine that they use different servers with different response times or treating members with older contracts different than members with newer contracts.
But for a solution can we not just drop the outer loop test? I think that people who got their currentWeather data in their buffer (like me) don't need it and people who still have an open connection (like you) might not need it too. Even if one has still an open connection the data will be all there while the parser is going through the stream (or not? Actually I am not 100% sure).
We have not found the answer to this anomalous behaviour. We have ideas for several probable causes. I still want to find an answer and I will test everything I can here on my side!

---

What's odd is the problem should be deterministic, it is the same code and (ESP) hardware, so why is it behaving differently, this is why I was focussing on network performance as it is the only thing that is variable. It is possible and I have seen this, that OWM data comes from different servers, as I currently have 3 of my designs running on the bench using identical code and yet often the displayed values especially weather forecasts are different! Values such as temperature and pressure are generally the same , so when I investigated deeper I saw different server addresses, it very subtle but it points to OWM (and the same of WU) not having or using identical software on their servers.

I would leave out the connected test as you have now and see how it performs. We don't know how much of the buffer / free RAM the streaming parser of ThingPulse consumes or how it streams/handshakes, but it looks like this is where the issue is likely to be especially when small amounts of data are being received. It never seems to go wrong with the forecast section for example.

---

Let's continue here: https://github.com/ThingPulse/esp8266-weather-station/issues/140