|
This page uses the New York Times Newswire API to generate AFSK audio. The Newswire API provides an up-to-the-minute stream of published articles. Press the appropriate Play AFSK button above. Ideally, this will load the audio player and start playing. In initial tests, Windows Google Chrome does not do the autoplay even if you set the site settings for sound to Always, but Android Chrome DOES autoplay.. Opera and Edge do autoplay. Browsers that do not autoplay (Chrome) will require you to hit the play icon in the player. In addition, playback will stop at the end of the content if the browser does not support autoplay. If the browser DOES support autoplay, at the end of playback, the player will be loaded with the latest content which will be played. Content is updated on the hour. If the played content ends before the top of the hour, the same content (which is the latest) will be loaded and played. If the playback started before the top of the hour and ended after the top of the hour, new content will be played. Both feeds are 170 Hz shift with 2125 Hz mark and 2295 Hz space. See RTTY FEEDS FOR YOUR MACHINES OR SOFTWARE for more TTY news feeds. |
This page uses 8 bit PCM wav files generated by a cron job once each hour. Cron calls a shell script that calls a series of PHP scripts. Here's the process.
The page makes use of the Times Newswire API. Curl fetches the json data from the API. PHP json_decode() converts the received json data to an array. The code then steps through each element in the results field and gets the article if the section is "World" or "U.S.". This restricts the feed to news articles removing opinion, puzzles, etc.
Each of the filtered results (filtered by section) is passed to a function that fetches the article. The URL for the article, the byline, the updated date, and the source (NY Times or NY TImes International) are passed to GetArticle().
GetArticle() fetches the artilce using curl. The resulting HTML is modified a bit to add CRCRLF in front of /p tags. The next step removes tags, so the CRCRLF serves as a paragraph break on the printer. CR is sent included twice to allow time for the carriage to return. Then the HTML is passed to a DOMDocument for parsing. The headline and article text are extracted using the code below. Because the article was fetched without logging in, there is often a string "We are having trouble retrieving the article content." before the end of the article. We terminate terminate the article at that point. Also, occasionally, some javascript gets into the article text. The article is terminated before the javascript.
function GetArticle($url, $author, $date, $source){
global $fh; // Where we are going to write stuff
$html = `curl {$url}`; // Get article html
$html=str_replace('</p>', "\r\n\n</p>", $html); // Add line breaks to paragraph breaks
$dom = new DOMDocument; //Create a new DOM document
$dom->loadHTML($html); //Parse the HTML.
// Find the headline
foreach($dom->getElementsByTagName('h1') as $record) {
if('headline' == $record->getAttribute('data-testid')){
$headline = $record->nodeValue;
}
}
// Find the article body
foreach($dom->getElementsByTagName('section') as $record) {
if('articleBody' == $record->getAttribute('name')){
$ArticleText = $record->nodeValue;
}
}
$EndPos=strpos($ArticleText,"We are having trouble retrieving the article content."); // Trim here to end
if($EndPos!=false) $ArticleText=substr($ArticleText,0,$EndPos); // Remove login request at end of article
$EndPos=strpos($ArticleText,"window.registerInteractive");
if($EndPos!=false) $ArticleText=substr($ArticleText,0,$EndPos); // Drop code that got into article body
$headline=wordwrap($headline,70); // Limit headline to 70 characers per line
$headline=str_replace("\n","\r\r\r\n",$headline); // Replace lf with crcrlf to give more time for carriage return
$ArticleText=str_replace("\xe2\x80\x99", "'", $ArticleText); // Replace UTF-8 apostorphe with apostrophe
$ArticleText=str_replace("\nImage","\nImage ",$ArticleText); // Add space after Image before caption
$ArticleText=wordwrap($ArticleText, 70); // Limit to 70 characters per line
$ArticleText=str_replace("\n","\r\r\r\n",$ArticleText); // Replace crlf with crcrlf to give more time for carriage return
if( (strlen($ArticleText)>70) ){// && (false!=strpos($ArticleText,"//")) ){ // Don't print empty articles or code comments
fwrite($fh,"\r\r\r\n----------------------------------------------------------\r\r\r\n\n"); // Article Separator
fwrite($fh,"$headline\r\r\r\n");
$byline = "$author, $date, $sourcce\r\r\r\r\n\n";
$byline = wordwrap($byline,70);
$byline=str_replace("\n","\r\r\r\n",$byline);
fwrite($fh,"$byline");
fwrite($fh,$ArticleText);
} // end don't print empty articles
}
The UTF-8 apostrophe is replaced with the ASCII apostrophe character. The headline and article are wordwrapped to 70 characters. Newlines are replaced with CRCRCRLF to provide time for the carriage return. Finally, the article is written to nyt.txt in the form of Headline [newline] author, date, source, and ArticleText.
A line of hyphens is put into nyt.txt between each article before the next article is appended.
The translation is based on an array where the index is the ASCII code and the value is the Baudot code. Since Baudot is only 5 bits, the 5 least significant bits are used to hold the Baudot code. The lsb holds the first bit to be transmitted (since UARTs transmit lsb first). The msb of the value is set if FIGS is required for this character (numbers and punctuation).
<?php
// Read ascii text file and create baudot text file
$AsciiToBaudotTable = array(
// Index is ASCII code (0..127). 5 LSB of data is Baudot code, transmitted
// LSB first. MSB is set if FIGS is required.
// See https://en.wikipedia.org/wiki/Baudot_code#ITA2 with lsb on right
0x00, // ASCII null. Send blank key
0x00, // ASCII SOH, send blank key
0x00, // ASCII STX, send blank key
0x00, // ASCII ETX, send blank key
0x00, // ASCII EOT
0x00, // ASCII ENQ. semd blank key
0x00, // ASCII ACK, send blank key
0x85, // ASCII Bell.FIGS S
0x00, // ASCII BS, send blank key
0x00, // ASCII HT, send blank key
0x02, // Line feed.
0x00, // ASCII VT, send blank key
0x00, // ASCII FF, send blank key
0x08, // Carriage return
...
0x04, // Space
0x8d, // Exclamation point, FIGS 0x0d
0x91, // Quote, FIGS 0x81
0x94, // #, FIGS 0x84
0x89, // $, FIGS 0x09
0x00, // ASCII percent sign, send blank key
0x9a, // &, FIGS 0x8a
0x8b, // Apostrophe, FIGS 0x0b
0x8f, // Open paren
0x92, // Close paren
0x00, // ASCII *, send blank key
0x00, // ASCII plus, send blank key
0x8c, // comma, FIGS 0x0c
0x83, // hyphen
0x9c, // period
0x9d, // slash
0x96, // zero
0x97, // one
0x93, // two
0x81, // three
0x8a, // four
0x90, // five
0x95, // six
...
0x99, // ? FIGS 0x18
0x00, // @ Send blank key
0x03, // A
0x19, // B
0x0e, // C
0x09, // D
0x01, // E
0x0d, // F
0x1a, // G
0x14, // H
0x06, // I
0x0b, // J
0x0f, // K
0x12, // L
0x1c, // M
0x0c, // N
0x18, // O
0x16, // P
0x17, // Q
0x0a, // R
0x05, // S
0x10, // T
0x07, // U
0x1e, // V
0x13, // W
0x1d, // X
0x15, // Y
0x11 // Z
);
function AsciiCharToBaudot($AsciiChar){
global $AsciiToBaudotTable;
$AsciiCode=ord($AsciiChar);
if($AsciiCode>0x60) $AsciiCode-=0x20; // Shift lower case to upper case
if($AsciiCode<91){ // Prevent running off end of array
return($AsciiToBaudotTable[$AsciiCode]);
}
}
function AsciiFileToBaudot($AsciiFileName, $BaudotFileName){
$AsciiFp=fopen($AsciiFileName, 'r'); // Open ascii file for read
$BaudotFp=fopen($BaudotFileName,'wb'); // Open baudot file for writing binary
if (!$AsciiFp) {
echo "Could not open file $AsciiFileName";
}
while (false !== ($AsciiChar = fgetc($AsciiFp))) { // Read character by character until eof
$BaudotChar=AsciiCharToBaudot($AsciiChar); // Translate to baudot
fwrite($BaudotFp,pack('c',$BaudotChar)); // Write to Baudot file
}
fclose($AsciiFp);
fclose($BaudotFp);
}
The resulting Baudot string is written to a file as packed 8 bit characters.
This script takes the file from above (text in Baudot) and converts it to a binary audio file. The file format is 8 bits per sample and 8,000 samples per second. 8 bits per sample was the standard for telephone circuits. See, for example, ISDN and T1. 8 bit 8 kHz is the lowest bitrate PCM WAV file available, enabling it to be easily played on web browsers. Further, with 8 bit samples, there is no worry about endianess (does the most significant or least significant byte come first?).
I first read about direct digital synthesis in an amateur radio magazine in the 1970s. I was immediately impressed with its simplicity and elegance. On each clock pulse, the latch captures the output of the adder. The latch output is sent back to one of the adder inputs. The other adder input is the frequency control. On each clock pulse, the latch captures the previous latch output plus the frequency control. The latch binary value would ramp up and wrap at a rate determined by the frequency control. The latch output is sent to a sine lookup table in ROM which changes the binary ramp to a binary sine. This drives the ADC for an analog sine wave of a user determined frequency.
DDS is even easier to do in software.
<?php
// Generate 8 bit PCM tone suitable for wav file (unsigned and biased up 128 for range 0 to 255).
$pi2=2.0*acos(-1.0); // Precalculate 2pi
$DdsRadiansPerSample=0;
function DdsFreqSet($freq){
// Set DdsRadiansPerSample so there is less math in DdsNextSample
global $DdsRadiansPerSample, $pi2;
$DdsRadiansPerSample=$freq*$pi2/8000.0;
}
function DdsNextSample(){
global $DdsRadiansPerSample, $pi2;
static $angle=0;
$angle+=$DdsRadiansPerSample;
while($angle>=$pi2) $angle-=$pi2;
return 128+(127*sin($angle));
}
function DdsWriteSamples($fh, $NumSamples){
while($NumSamples>0){
fwrite($fh,pack("C",DdsNextSample()));
$NumSamples--;
}
}
/*
// Test
$fh=fopen("tone.bin","wb"); // Open a file to write in binary
DdsFreqSet(1000); // Set frequency to 1 kHz
DdsWriteSamples($fh,80000); // Write 10 seconds of tone
fclose($fh);
*/
?>
The code is pretty straightforward and heavily commented. One thing to note is the return value of DdsNextSample(). First off, the angle is set to wrap around at 2*pi so it does not increase forever. Second, we use the sin calculation instead of a lookup table. This is simpler, provides more accuracy, and avoids the need for some big table. Finally, note that the returned value is 128+(127*sin()). Most PCM WAV files use signed integer sample values. However, 8 bit WAV uses unsigned integers with 0 representing the negative peak of the waveform, and 255 representing the positive peak of the waveform.
Note DdsWriteSamples() at the end of the code above. This writes the specified number of samples to the file specified by $fh. That, along with DdsFreqSet() allows us to very simply add tone of a desired frequency for a desired number of samples. For example, we could set the frequency to 2125 Hz and the number of samples to 8000/BR, where BR is the baud rate, to write one bit time of mark.
In this section of the code, we use the above DdsFreqSet() and DdsWriteSamples() to write the tones for mark and space data bits and the longer stop bit. That code is shown below.
function MarkBit($fp){
// Send SamplesPerBit of 2125 Hz to file
global $SamplesPerBit;
DdsFreqSet(2125); // Mark tone frequency
DdsWriteSamples($fp,$SamplesPerBit); // 8 samples per millisecond
}
function SpaceBit($fp){
// Send SamplesPerBits of 2295 Hz to file
global $SamplesPerBit;
DdsFreqSet(2295); // Space tone frequency
DdsWriteSamples($fp,$SamplesPerBit); // 8 samples per millisecond
}
function StopBit($fp){
// Send SamplesPerStop of 2125 Hz to file
global $SamplesPerStop;
DdsFreqSet(2125); // Mark tone frequency
DdsWriteSamples($fp,$SamplesPerStop); // 8 samples per millisecond
}
$SamplesPerBit and $SamplesPerStop depend on the baud rate and are set in code below. Otherwise, this is just set a frequency and write samples!
The code below is similar to "bit banging" serial output with software. Here, however, we do not need to insert delays for bit times. Instead, we just write the appropriate number of samples. Here's the code to send the LTRS character and the FIGS character.
function LtrsToAfsk($AudioFp){
$BaudotChar=0b11111;
SpaceBit($AudioFp); // Start bit
$BitMask=1;
do{
if(($BitMask & $BaudotChar)==0){
SpaceBit($AudioFp); // Bit is low, send space
}else{
MarkBit($AudioFp); // Bit is high, send space
}
$BitMask=$BitMask<<1; // Check next most significant bit next time
}while($BitMask<0b100000);
StopBit($AudioFp);
}
function FigsToAfsk($AudioFp){
$BaudotChar=0b11011;
SpaceBit($AudioFp); // Start bit
$BitMask=1;
do{
if(($BitMask & $BaudotChar)==0){
SpaceBit($AudioFp); // Bit is low, send space
}else{
MarkBit($AudioFp); // Bit is high, send space
}
$BitMask=$BitMask<<1; // Check next most significant bit next time
}while($BitMask<0b100000);
StopBit($AudioFp);
}
The start bit is sent, then a bitmask is set with the lsb high. This is bitwise anded with the Baudot character to determine whether a mark or space tone should be appended to the file. The bitmask is then shifted left one bit and the process repeated. When the bitmask has been shifted five times, the loop is exited and the stop bit is appended to the audio file. Separate functions are used for LTRS and FIGS since the code below has to insert these into the audio stream.
The code below is largely the same as that above, but it inserts any needed FIGS or LTRS based on whether the msb of the 8 bit Baudot character (Baudot in the 5 lsb, msb set if FIGS required) is set and whether the last character was in the FIGS or LTRS set.
function BaudotCharToAfsk($BaudotChar, $AudioFp){
static $figs=0; // Remember whether we are shifted to figs or not
if(($BaudotChar>0x7f)&&($figs==0)){
FigsToAfsk($AudioFp); // Send figs
$figs=1; // remember we are in figs
}else{
if(($BaudotChar<0x80)&&($figs==1)){ // character not figs but we are in figs
LtrsToAfsk($AudioFp); // Send Ltrs
$figs=0; // Not in figs anymore
}
}
SpaceBit($AudioFp); // Send start bit
$BitMask=1;
do{
if(($BitMask & $BaudotChar)==0){
SpaceBit($AudioFp); // Bit is low, send space
}else{
MarkBit($AudioFp); // Bit is high, send space
}
$BitMask=$BitMask<<1; // Check next most significant bit next time
}while($BitMask<0b100000);
StopBit($AudioFp);
}
Now that we can write an AFSK character to a file, we need to create the audio file, open the Baudot text file, and feed the characters to BaudotCharToAfsk(). This is handled by the code below.
function BaudotFileToAfsk($BaudotFileName, $AfskFileName){
$BaudotFp=fopen($BaudotFileName, 'r'); // Open baudot file for read
$AfskFp=fopen($AfskFileName,'wb'); // Open afsk file for writing binary
if (!$BaudotFp) {
echo "Could not open file $BaudotFileName";
}
DdsFreqSet(2125);
DdsWriteSamples($AfskFp, 80000); // 10 seconds of mark at start
while (false !== ($BaudotChar = fgetc($BaudotFp))) { // Read character by character until eof
BaudotCharToAfsk(ord($BaudotChar),$AfskFp);
}
fclose($BaudotFp);
fclose($AfskFp);
}
As mentioned above, we need to determine how many audio samples per data bit and stop bit. That is handled by the code below. The words per minute value is passed to the script as a command line argument. For simplcity, the stop bit length is set to 1.5 times the data bit length. This results in a stop bit length of 33 ms for 60 wpm instead of the normal 31 ms. This is close enough and easier to calculate.
switch($argv[1]){ // Get wpm from command line
default:
case 60:
$SamplesPerBit=8000/45.45; // 45.45 bps
$SamplesPerStop=1.5*$SamplesPerBit;
echo "Creating 60 wpm nyt.bin\r\n";
break;
case 100:
$SamplesPerBit=8000/75; // 75 bps
$SamplesPerStop=1.5*$SamplesPerBit;
echo "Creating 100 wpm nyt.bin\r\n";
break;
}
Now that we've created an 8 bit PCM file of the audio, we need to put a WAV header on the front so a player knows what to do with it. The fairly self explanatory code is below.
<?php
// References
// https://www.maztr.com/audiofileanalyzer - Audio file analyzer
// https://homspace.nl/samplerbox/WAVE%20File%20Format.htm - Wave file description
function BinToWav($BinFileName, $WavFileName){
// Input a binary 8 bit audio file (unsigned 8 bit samples, 8 kHz sample rate).
// Write a WAV file by adding the WAV header
$BinSize=filesize($BinFileName); // How many bytes in the binary file
$fhw=fopen("/home/harold/nyt/data/WavHeader.bin","wb"); // Get file handle for write
fwrite($fhw,"RIFF"); // RIFF block id
fwrite($fhw,pack("V",$BinSize+36)); // RIFF size (binary data size + 36) as 32 bit little endian
fwrite($fhw,"WAVEfmt "); // WAVE block id and fmt block id
fwrite($fhw,pack("V",16)); // fmt block size as 32 bit little endian
fwrite($fhw,pack("v",1)); // 01=PCM as 16 bit little endian
fwrite($fhw,pack("v",1)); // 01 channels as 16 bit little endian
fwrite($fhw,pack("V",8000)); // 8000 samples per second as 32 bit little endian
fwrite($fhw,pack("V",8000)); // 8000 bytes per second as 32 bit little endian
fwrite($fhw,pack("v",1)); // block align = 01 as 16 bit little endian
fwrite($fhw,pack("v",8)); // 8 bits per sample as 16 bit little endian
fwrite($fhw,"data"); // data block id
fwrite($fhw,pack("V",$BinSize)); // size of following binary
fclose($fhw); // close file after creating header
`cat /home/harold/nyt/data/WavHeader.bin {$BinFileName} > {$WavFileName}`;
}
$TimeStamp= time();
$DestName = "/home/harold/nyt/data/nyt_$TimeStamp.wav";
BinToWav("/home/harold/nyt/data/nyt.bin", $DestName);
?>
Most of the code generates the WAV header. The 8 bit audio file is then concatenated to the WAV header and written to an output file. Note that a timestamp is added to the file so each time this is run, a new file is created instead of overwriting the old file.
10 60 wpm WAV files are kept, and 10 100 wpm WAV files. As noted above, they have the timestamp appended to their names. We can do a simple sort and delete the oldest files. This is done in the code below.
<?php
// Delete old wav files in public_html
// Sort files descending, latest on top. Do 100 wpm first
$FileList=scandir("/home/harold/public_html/org/w6iwi/rtty/audio/nyt/audio/100wpm",SCANDIR_SORT_DESCENDING);
$count=count($FileList); // how many files found
for($n=10;$n<$count-4;$n++){ // Start deleting at file 10. Stop before latest, . , and ..
$pos=strpos($FileList[$n],"nyt"); // See if filename starts with nyt
if(0==$pos){ // If so, delete file
echo "Deleting /home/harold/public_html/org/w6iwi/rtty/audio/nyt/audio/100wpm/$FileList[$n]\r\n";
unlink("/home/harold/public_html/org/w6iwi/rtty/audio/nyt/audio/100wpm/$FileList[$n]");
}
}
// Now do 60 wpm
$FileList=scandir("/home/harold/public_html/org/w6iwi/rtty/audio/nyt/audio/60wpm",SCANDIR_SORT_DESCENDING);
$count=count($FileList); // how many files found
for($n=10;$n<$count-4;$n++){ // Start deleting at file 10. Stop before latest, . , and ..
$pos=strpos($FileList[$n],"nyt"); // See if filename starts with nyt
if(0==$pos){ // If so, delete file
echo "Deleting /home/harold/public_html/org/w6iwi/rtty/audio/nyt/audio/60wpm/$FileList[$n]\r\n";
unlink("/home/harold/public_html/org/w6iwi/rtty/audio/nyt/audio/60wpm/$FileList[$n]");
}
}
?>
Originally, I just had an update rewrite the wav file. But, if the file was being played, this resulted in a corrupted file. Often the WAV header that was present when playback started was longer than the file when playback ended resulting in the player stalling due to a lack of data. As noted above, now each time a different WAV file is written with the name including the timestamp of when it was created. In each of the audio directories is the below code (latest.php).
<?php
// Redirect to latest wav file.
$FileList=scandir(".",SCANDIR_SORT_DESCENDING); // Sort files descending, latest on top
$url="$FileList[0]"; // Make this the url and redirect
header("Location: $url"); // redirect
?>
<html>
<head>
<title>Redirect to $url</title>
<META HTTP-EQUIV="REFRESH" CONTENT="1; URL=" >
</head>
<body>
Redirect to <a href="<?php echo("$url\">$url");?>
</body>
</html>
The URL the 60 wpm audio player uses for its content is audio/60wpm/latest.php?n=' + currentAudioIndex . As shown above, latest.php redirects the browser to the most recent wav file. That file is not deleted until 9 more files have been written. Each file typically takes 45 minutes to play and will not be deleted for 10 hours. When the player requests the latest file, it may be pointed to the same file if a new one has not been created, or to a new one if it has (it's always the latest). In addition, a query string (?n=currentAudioIndex) is added to the URL so the browser does not attempt to play a cached file.
More than you ever wanted to know!
Comments to harold@w6iwi.org