Biligrab 0.3 linux与OSX下自动抓B站视频和弹幕的东西（更新：修复重要BUG！！）

B站有自己的视频CDN了，咱们加进来。

对于海外有时比sina的蓝汛慢，但稳定。

差点break错了尼玛死。。。怪不得professor不让用呢。

更新：还是break错了。。。。。。。。。。。这个东西尽可能别用啊。

MIT，爱干啥干啥。

顺手扔上Gist，方便改：https://gist.github.com/superwbd/9605757

代码藏起来，否则主页太慢。

'''
Biligrab 0.3
Beining@ACICFG
cnbeining[at]gmail.com
MIT licence
'''

import sys
import os
from StringIO import StringIO
import gzip
import urllib2
import sys

reload(sys)
sys.setdefaultencoding('utf-8')

def main(vid, p, oversea):
    cid = 0
    title = ''
    partname = ''
    biliurl = 'http://api.bilibili.tv/view?type=xml&appkey=876fe0ebd0e67a0f&id=' + str(vid) + '&page=' + str(p)
    videourl = 'http://www.bilibili.tv/video/av'+vid+'/index_'+p+'.html'
    print('Fetching webpage...')
    request = urllib2.Request(biliurl)
    response = urllib2.urlopen(request)
    data = response.read()
    data_list = data.split('\n')
    for lines in data_list:
        if 'cid' in lines:
            cid = lines[7:-6]
            print('cid is ' + str(cid))
        if 'partname' in lines:
            partname = lines[12:-11]
            print('partname is ' + str(partname))
        if 'title' in lines:
            title = lines[9:-8]
            print('title is ' + str(title))
    if cid is 0:
        print('Cannot find cid, trying to do it brutely...')
        print('Fetching webpage...')
        request = urllib2.Request(videourl)
        request.add_header('Accept-encoding', 'gzip')
        response = urllib2.urlopen(request)
        if response.info().get('Content-Encoding') == 'gzip':
            buf = StringIO( response.read())
            f = gzip.GzipFile(fileobj=buf)
            data = f.read()
        data_list = data.split('\n')
        #Todo: read title
        for lines in data_list:
            if 'cid=' in lines:
                cid = lines.split('&')
                cid = cid[0].split('=')
                cid = cid[-1]
                print('cid is ' + str(cid))
                break
    if cid is 0:
        print('Cannot get cid anyway, you fix it, I am off to get some coffee...')
        exit()
    #start to make folders...
    if title is not '':
        folder = title
    else:
        folder = cid
    if partname is not '':
        filename = partname
    elif title is not '':
        filename = title
    else:
        filename = cid
    folder_to_make = os.getcwd() + '/' + folder
    if not os.path.exists(folder_to_make):
        os.makedirs(folder_to_make)
    os.chdir(folder_to_make)
    print('Fetching XML...')
    os.system('curl -o "'+filename+'.xml" --compressed  http://comment.bilibili.cn/'+cid+'.xml')
    #os.system('gzip -d '+cid+'.xml.gz')
    print('The XML file, ' + filename + '.xml should be ready...enjoy!')
    print('Finding video location...')
    #try api
    if oversea == '1':
        request = urllib2.Request('http://interface.bilibili.cn/v_cdn_play?cid='+cid, headers={ 'User-Agent' : 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36', 'Cache-Control': 'no-cache', 'Pragma': 'no-cache' })
    else:
        request = urllib2.Request('http://interface.bilibili.tv/playurl?cid='+cid, headers={ 'User-Agent' : 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36', 'Cache-Control': 'no-cache', 'Pragma': 'no-cache' })
    response = urllib2.urlopen(request)
    data = response.read()
    data_list = data.split('\r')
    #print(data_list)
    rawurl = []
    vid_num = 0
    for lines in data_list:
        lines = str(lines)
        if '<url>' in lines:
            if 'youku'  in lines:
                url = lines[17:-9]
            elif 'sina' in lines:
                url = lines[16:-9]
            elif 'qq.com' in lines:
                url = lines[17:-9]
            elif 'letv.com' in lines:
                url = lines[17:-9]
                break
            elif 'acgvideo' in lines:
                url = lines[17:-9]
            rawurl.append(url)
        if 'backup_url' in lines:
            break
    if rawurl is []:  #hope this never happen
        request = urllib2.Request('http://www.flvcd.com/parse.php?kw='+videourl)
        request.add_header('Accept-encoding', 'gzip')
        response = urllib2.urlopen(request)
        data = response.read()
        data_list = data.split('\n')
        for items in data_list:
            if 'name' in items and 'inf' in items and 'input' in items:
                c = items
                rawurl = c[39:-5]
                rawurl = rawurl.split('|')
                break
    #print(rawurl)
    vid_num = len(rawurl)
    #print(rawurl)
    print(str(vid_num) + ' videos to download, fetch yourself a cup of coffee...')
    for i in range(vid_num):
        print('Downloading ' + str(i+1) + ' of ' + str(vid_num) + ' videos...')
        #print('aria2c -llog.txt -c -s16 -x16 -k1M --out '+str(i)+'.flv "'+rawurl[i]+'"')
        os.system('aria2c -larialog.txt -c -s16 -x16 -k1M --out '+str(i)+'.flv "'+rawurl[i]+'"')
    f = open('ff.txt', 'w')
    ff = ''
    os.getcwd()
    for i in range(vid_num):
        ff = ff + 'file \'' + str(os.getcwd()) + '/'+ str(i) + '.flv\'\n'
    ff = ff.encode("utf8")
    f.write(ff)
    f.close()
    print('Concating videos...')
    os.system('ffmpeg -f concat -i ff.txt -c copy "'+filename+'".mp4')
    os.system('rm -r ff.txt')
    for i in range(vid_num):
        os.system('rm -r '+str(i)+'.flv')
    print('Done, enjoy yourself!')
    exit()

vid = str(input('av'))
p = str(input('P'))
oversea = str(input('Oversea?'))
main(vid, p, oversea)

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

'''

Biligrab 0.3

Beining@ACICFG

cnbeining[at]gmail.com

MIT licence

'''

import sys

import os

from StringIO import StringIO

import gzip

import urllib2

import sys

reload(sys)

sys.setdefaultencoding('utf-8')

def main(vid, p, oversea):

cid = 0

title = ''

partname = ''

biliurl = 'http://api.bilibili.tv/view?type=xml&appkey=876fe0ebd0e67a0f&id=' + str(vid) + '&page=' + str(p)

videourl = 'http://www.bilibili.tv/video/av'+vid+'/index_'+p+'.html'

print('Fetching webpage...')

request = urllib2.Request(biliurl)

response = urllib2.urlopen(request)

data = response.read()

data_list = data.split('\n')

for lines in data_list:

if 'cid' in lines:

cid = lines[7:-6]

print('cid is ' + str(cid))

if 'partname' in lines:

partname = lines[12:-11]

print('partname is ' + str(partname))

if 'title' in lines:

title = lines[9:-8]

print('title is ' + str(title))

if cid is 0:

print('Cannot find cid, trying to do it brutely...')

print('Fetching webpage...')

request = urllib2.Request(videourl)

request.add_header('Accept-encoding', 'gzip')

response = urllib2.urlopen(request)

if response.info().get('Content-Encoding') == 'gzip':

buf = StringIO( response.read())

f = gzip.GzipFile(fileobj=buf)

data = f.read()

data_list = data.split('\n')

#Todo: read title

for lines in data_list:

if 'cid=' in lines:

cid = lines.split('&')

cid = cid[0].split('=')

cid = cid[-1]

print('cid is ' + str(cid))

break

if cid is 0:

print('Cannot get cid anyway, you fix it, I am off to get some coffee...')

exit()

#start to make folders...

if title is not '':

folder = title

else:

folder = cid

if partname is not '':

filename = partname

elif title is not '':

filename = title

else:

filename = cid

folder_to_make = os.getcwd() + '/' + folder

if not os.path.exists(folder_to_make):

os.makedirs(folder_to_make)

os.chdir(folder_to_make)

print('Fetching XML...')

os.system('curl -o "'+filename+'.xml" --compressed http://comment.bilibili.cn/'+cid+'.xml')

#os.system('gzip -d '+cid+'.xml.gz')

print('The XML file, ' + filename + '.xml should be ready...enjoy!')

print('Finding video location...')

#try api

if oversea == '1':

request = urllib2.Request('http://interface.bilibili.cn/v_cdn_play?cid='+cid, headers={ 'User-Agent' : 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36', 'Cache-Control': 'no-cache', 'Pragma': 'no-cache' })

else:

request = urllib2.Request('http://interface.bilibili.tv/playurl?cid='+cid, headers={ 'User-Agent' : 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36', 'Cache-Control': 'no-cache', 'Pragma': 'no-cache' })

response = urllib2.urlopen(request)

data = response.read()

data_list = data.split('\r')

#print(data_list)

rawurl = []

vid_num = 0

for lines in data_list:

lines = str(lines)

if '<url>' in lines:

if 'youku' in lines:

url = lines[17:-9]

elif 'sina' in lines:

url = lines[16:-9]

elif 'qq.com' in lines:

url = lines[17:-9]

elif 'letv.com' in lines:

url = lines[17:-9]

break

elif 'acgvideo' in lines:

url = lines[17:-9]

rawurl.append(url)

if 'backup_url' in lines:

break

if rawurl is []: #hope this never happen

request = urllib2.Request('http://www.flvcd.com/parse.php?kw='+videourl)

request.add_header('Accept-encoding', 'gzip')

response = urllib2.urlopen(request)

data = response.read()

data_list = data.split('\n')

for items in data_list:

if 'name' in items and 'inf' in items and 'input' in items:

c = items

rawurl = c[39:-5]

rawurl = rawurl.split('|')

break

#print(rawurl)

vid_num = len(rawurl)

#print(rawurl)

print(str(vid_num) + ' videos to download, fetch yourself a cup of coffee...')

for i in range(vid_num):

print('Downloading ' + str(i+1) + ' of ' + str(vid_num) + ' videos...')

#print('aria2c -llog.txt -c -s16 -x16 -k1M --out '+str(i)+'.flv "'+rawurl[i]+'"')

os.system('aria2c -larialog.txt -c -s16 -x16 -k1M --out '+str(i)+'.flv "'+rawurl[i]+'"')

f = open('ff.txt', 'w')

ff = ''

os.getcwd()

for i in range(vid_num):

ff = ff + 'file \'' + str(os.getcwd()) + '/'+ str(i) + '.flv\'\n'

ff = ff.encode("utf8")

f.write(ff)

f.close()

print('Concating videos...')

os.system('ffmpeg -f concat -i ff.txt -c copy "'+filename+'".mp4')

os.system('rm -r ff.txt')

for i in range(vid_num):

os.system('rm -r '+str(i)+'.flv')

print('Done, enjoy yourself!')

exit()

vid = str(input('av'))

p = str(input('P'))

oversea = str(input('Oversea?'))

main(vid, p, oversea)

哦，你把全文看完了？

ffmpeg有3种Concat方法。我们用了最笨的。

因为别的方法都不支持H.264的不转码Concat。如果你使用MPEG，可以玩玩，那个的实现简单的多。

8 thoughts on “Biligrab 0.3 linux与OSX下自动抓B站视频和弹幕的东西（更新：修复重要BUG！！）”

akw28888 2014 年 3 月 30 日 at am 10:06

B站CDN現在完全沒速度永遠卡在那邊

Reply ↓
1. Beining Post author2014 年 3 月 30 日 at pm 4:49
  
  看来我得澄清一下。。。
  不是说，在Biligrab里面写“Oversea=1”，就一点会得到海外源。这个功能只是说，会使用B站的CDN的API进行解析，相当于你点了播放器的“视频源加速”。如果有自建CDN，那不排除会返回B站CDN，如果没有，会返回原始的源。
  当然这个我会再看一下。。。B站的海外CDN也不少，是美国的那个FDC有问题还是荷兰的FDC有问题？还是国内的哪台CDN呢？
  
  Reply ↓
  1. akw28888 2014 年 3 月 31 日 at am 10:35
    
    菊巨的軟件我都還沒測試過
    我是本地在看視頻時候
    B站那邊會永遠解析到中國內CDN節點
    有30分之1的機率會走新浪節點
    然後永遠不可能走荷蘭跟chicago那兩台………
    
    Reply ↓
    1. Beining Post author2014 年 4 月 1 日 at am 4:29
      
      从IP来看，是hinet的IP呢。。
      因为我自己的IP段已经不在东亚好多年了（目前常出现的IP在美西和加东），所以也没啥好办法查到那边的CDN策略。。。
      那么，可以感觉到，在大陆乃至整个东南亚，Bilibili的确在大量使用自己的CDN以减轻sina的负担，免得sina寄刀片到杭州去。
      北美部分的CDN使用很少，即使是版权有问题的东西（难道是故意把海外版权问题扔给sina？）。当然极其偶尔也是会出现海外节点的。
      多说一句，这样看来，B站的运营成本现在已经不是特别低了。按理说可以推算出来，但是我没这个胆量用API大量抓取，会误伤别的软件的。。。也没那个能力啊。
      对于台湾来说，国际出口实在太少，如果使用北美乃至欧洲源就太不明智了。至少在这个大是大非上 Bilibili的CDN没有出错。当然会不会（或者能不能）根据具体的peering情况继续优化就不得而知了。。。大陆的一切peering信息都是天机不可泄露，不爽啊。
      另：咱比较懒，如果不更新会很长时间不更，一想起来有可能一天发一堆。。。总之最好用新版，增加了点小功能，防止报错啥的wwwww
      
      Reply ↓
StarBrilliant 2014 年 6 月 14 日 at am 11:06

请帮忙测试 OS X 下的兼容性 https://github.com/m13253/biligrab-danmaku2ass

Reply ↓
1. Beining Post author2014 年 6 月 14 日 at pm 11:18
  
  mpv brew不上悲催。。。。。
  
  Reply ↓
  1. StarBrilliant 2014 年 6 月 14 日 at pm 11:36
    
    我一分钟前才 brew 上 mpv。
    你看官网 mpv.io 下载页面的指示嘛。
    不过我现在 Python 3 brew 不上……
    
    Reply ↓
    1. Beining Post author2014 年 6 月 15 日 at pm 10:10
      
      貌似还是不吃。。。悲催
      
      Reply ↓

Beining 's Another site

by Beining@ACI-CFG V8

Biligrab 0.3 linux与OSX下自动抓B站视频和弹幕的东西（更新：修复重要BUG！！）

8 thoughts on “Biligrab 0.3 linux与OSX下自动抓B站视频和弹幕的东西（更新：修复重要BUG！！）”

Leave a Reply Cancel reply